Tunnel segment positioning and installation method and system based on image recognition

Abstract

The embodiment of the present application provides a tunnel segment positioning and installation method and system based on image recognition, three-dimensional scanning images of a tunnel segment assembling surface are collected through a three-dimensional scanning device arranged at the end of a mechanical arm, and the position of a first target area including a target assembling opening is determined. According to the position of the first target area, the end of the mechanical arm is moved to a visual image collection position corresponding to the first target area. Plane visual images and three-dimensional visual images of the first target area are collected through a visual device arranged at the end of the mechanical arm. According to the plane visual images and the three-dimensional visual images, three-dimensional space data of the target assembling opening is obtained, and based on the three-dimensional space data of the target assembling opening, the mechanical arm is controlled to perform an assembling operation of a tunnel segment to be assembled. The method can improve the positioning and installation accuracy and efficiency of the tunnel segment.

Description

Technical Field

[0001] This application relates to the field of tunnel construction technology, and more specifically, to a method and system for positioning and installing tunnel segments based on image recognition. Background Technology

[0002] With the rapid development of the underground rail transit industry, urban subways are reaching saturation, leading to a rapid increase in mountain and river-crossing tunnels. Mountain and river-crossing tunnels present significant construction challenges, characterized by high overburden, large-scale excavation, and complex geological conditions. Conventional precast tunnel segment support methods are insufficient to guarantee tunnel construction quality. Therefore, it is necessary to add tunnel segment structural components to enhance the assembly strength of the tunnel segments and ensure the quality of the tunnel ring formation. Currently, conventional tunnel segment structural component assembly operations mainly rely on remote control using robotic arms. However, this method suffers from low assembly accuracy, and during actual operation, obstructed views make it difficult for operators to accurately observe the splicing joints and the specific position of the robotic arm, resulting in low efficiency and poor accuracy in splicing positioning.

[0003] Therefore, improving the accuracy and efficiency of locating and installing tunnel segments during tunnel construction is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and system for positioning and installing tunnel segments based on image recognition, so as to improve the accuracy and efficiency of positioning and installing tunnel segments during tunnel construction.

[0005] In a first aspect, this application provides a method for positioning and installing tunnel segments based on image recognition, including: The location of the first target area, including the target assembly opening, is determined by acquiring three-dimensional scan images of the tunnel segment assembly surface using a three-dimensional scanning device deployed at the end of a robotic arm. Based on the location of the first target area, move the end of the robotic arm to the visual image acquisition position corresponding to the first target area; The planar visual image and the three-dimensional visual image of the first target area are acquired by a vision device deployed at the end of the robotic arm. The accuracy of the planar visual image and the three-dimensional visual image is higher than that of the three-dimensional scanned image. The installation position of the vision device at the end of the robotic arm is lower than that of the three-dimensional scanned image at the end of the robotic arm. The field of view of the vision device is smaller than that of the three-dimensional scanned image. Based on the planar visual image and the three-dimensional visual image, the three-dimensional spatial data of the target assembly port is obtained; Based on the three-dimensional spatial data of the target assembly port, the robotic arm is controlled to perform the assembly operation of the tunnel segments to be assembled.

[0006] Secondly, this application provides a tunnel segment positioning and installation system based on image recognition. The tunnel segment positioning and installation system based on image recognition includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the tunnel segment positioning and installation system based on image recognition implements the aforementioned tunnel segment positioning and installation method based on image recognition.

[0007] The tunnel segment positioning and installation method and system based on image recognition provided in this application first uses a 3D scanning device deployed at the end of a robotic arm to acquire 3D scan images of the tunnel segment assembly surface, determining the location of a first target area containing the target assembly opening. Then, based on this location, the end of the robotic arm is moved to the corresponding visual image acquisition position. Next, a visual device deployed at the end of the robotic arm, installed at a lower position with a smaller field of view but higher precision, acquires planar and 3D visual images of the first target area. Based on these images, 3D spatial data of the target assembly opening is obtained. Finally, based on this 3D spatial data, the robotic arm is controlled to perform the assembly operation of the tunnel segment to be assembled, thereby achieving more precise positioning and installation of the tunnel segment, improving the quality and efficiency of tunnel segment assembly, reducing assembly errors, ensuring the safety and stability of tunnel construction, and thus providing a strong guarantee for the high-quality construction of tunnel projects. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0009] Figure 1 A schematic flowchart illustrating a tunnel segment positioning and installation method based on image recognition, provided for an embodiment of this application; Figure 2 This is a schematic diagram of a tunnel segment positioning and installation system based on image recognition, provided in an embodiment of this application.

[0010] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Figure 1 This is a flowchart illustrating a tunnel segment positioning and installation method based on image recognition, provided as an embodiment of this application. It should be understood that in other embodiments, the order of some steps in the tunnel segment positioning and installation method based on image recognition in this embodiment can be shared according to actual needs, or some steps can be omitted or maintained. Figure 1 As shown, the method may include the following steps: Step S110: Acquire a three-dimensional scan image of the tunnel segment assembly surface using a three-dimensional scanning device deployed at the end of the robotic arm, and determine the location of the first target area, including the target assembly opening.

[0013] In the actual operation of tunnel segment positioning and installation, in order to accurately locate the target assembly joint, a 3D scanning device installed at the end of a robotic arm is first used to scan the tunnel segment assembly surface. The 3D scanning device has high-precision spatial data acquisition capabilities, enabling it to capture the complex spatial information of the tunnel segment assembly surface and generate a 3D scan image. The 3D scan image not only includes the geometry of the tunnel segment surface but also its position and orientation information in three-dimensional space. The target assembly joint can be, for example, a "D-shaped" assembly joint with a pre-reserved inner groove at the longitudinal joint of the tunnel segment, through which adjacent tunnel segments are joined.

[0014] During the scanning process, the 3D scanning equipment emits specific scanning signals, such as laser beams or structured light. These signals interact with the tunnel segment assembly surface and are reflected back. The 3D scanning equipment calculates the 3D coordinates of each point on the tunnel segment surface based on the characteristics of the reflected signals. To ensure the comprehensiveness and accuracy of the scan, the 3D scanning equipment can operate according to certain scanning modes, such as scanning the assembly surface from multiple angles and directions to obtain complete 3D information.

[0015] After obtaining the 3D scan image, it is necessary to determine the location of the first target area containing the target assembly port. The target assembly port is the location where the tunnel segment to be assembled needs to be installed, and accurately determining its location is crucial for subsequent assembly operations. This process requires comprehensive consideration of various features and information in the image, achieved through a series of image processing and analysis algorithms.

[0016] Step S111: Perform image feature extraction processing on the three-dimensional scanned image to obtain composite spatial features included in the three-dimensional scanned image. The composite spatial features include at least one of edge contour features, geometric shape features, and surface texture features.

[0017] Image feature extraction from 3D scanned images is a crucial step in determining the location of the primary target region. Image feature extraction aims to extract representative and discriminative features from complex 3D scanned images for subsequent analysis and matching.

[0018] For edge contour feature extraction, gradient-based edge detection algorithms can be used. In 3D scanned images, the edges of tunnel segments typically exhibit abrupt changes in pixel values. By calculating the gradient values ​​at each point in the image, the location of the edges can be detected. For example, the Sobel operator or Canny edge detection algorithm can be used to calculate the gradients in the horizontal and vertical directions of the 3D scanned image. Then, points with larger gradient values ​​are selected based on a preset threshold; these points constitute the edge contours of the tunnel segments. Edge contour features can clearly delineate the shape of the tunnel segments, providing a foundation for subsequent shape analysis and matching.

[0019] Geometric feature extraction focuses on analyzing the overall shape and structure of tunnel segments. For example, the shape can be described by calculating geometric parameters such as the area, perimeter, and aspect ratio of the tunnel segments. Furthermore, shape descriptors, such as Hu moments, can be used to quantify the shape of tunnel segments. Hu moments are rotationally, translationally, and scaling-invariant, effectively describing the shape characteristics of tunnel segments. Even if the position, orientation, or size of the tunnel segment changes in the 3D scan image, the value of its Hu moment remains relatively stable.

[0020] Surface texture features reflect the microstructure and details of the tunnel segment surface. Gray-level co-occurrence matrix (GLCM) can be used to extract these features. The GLCM describes the spatial relationships between different gray levels in a 3D scanned image. By calculating its statistical characteristics, such as contrast, correlation, energy, and homogeneity, the texture features of the tunnel segment surface can be obtained. These texture features can help distinguish different types of tunnel segments and identify damage or defects on the tunnel segment surface.

[0021] In practice, one or more composite spatial features can be extracted depending on the specific circumstances. For example, if the edge contour of the tunnel segment is relatively clear, the edge contour features can be extracted; if the shape of the tunnel segment has obvious features, the geometric shape features can be extracted; if the texture information on the surface of the tunnel segment can help distinguish different tunnel segments or identify the target assembly port, the surface texture features can be extracted.

[0022] Step S112: Perform feature matching processing on the composite spatial features and the spatial structural features of the target assembly port to determine whether the three-dimensional scan image includes the first target region.

[0023] After obtaining the composite spatial features of the 3D scanned image, it is necessary to perform feature matching processing with the spatial structural features of the target assembly port to determine whether the 3D scanned image contains the first target region. The spatial structural features of the target assembly port are known in advance, such as through design drawings or historical measurement data.

[0024] Feature matching can employ various methods, such as feature point-based matching or feature descriptor-based matching. Feature point-based matching first detects feature points in the spatial structural features of the 3D scanned image and the target assembly port, then calculates descriptors for these feature points, and finally performs matching by comparing the similarity of the descriptors. For example, Scale Invariant Feature Transform (SIFT) or Speed-Up Robust Feature Transform (SURF) algorithms can be used to detect feature points and calculate their feature descriptors. Then, nearest neighbor matching or Random Sample Consensus (RANSAC) algorithms are used to filter out matching feature point pairs.

[0025] Feature descriptor-based matching methods directly compare descriptors of composite spatial features and spatial structural features of the target assembly opening. For example, for edge contour features, shape context descriptors can be used to describe their shape, and then the similarity between two descriptors is determined by calculating the distance between them. If the distance is less than a preset threshold, the two features are considered to match.

[0026] To improve the accuracy and reliability of the matching process, a multi-feature fusion method can be employed. This involves simultaneously considering multiple features such as edge contour features, geometric shape features, and surface texture features to comprehensively determine whether a region matching the target assembly port exists in the 3D scanned image. For example, different weights can be assigned to different features, and then a weighted similarity score can be calculated. If the score exceeds a preset threshold, the 3D scanned image is considered to contain the first target region.

[0027] Step S113: If the first target region is not included in the three-dimensional scan image, control the movement and repeated scanning of the robotic arm based on a preset time interval and a preset scanning path until the first target region is included in the three-dimensional scan image.

[0028] When the 3D scan image does not contain the first target region, the robotic arm needs to be controlled to move and repeat the scan. The preset time interval and preset scan path are pre-set parameters used to guide the robotic arm's movement.

[0029] The preset time interval determines the time interval between each movement of the robotic arm, and its setting needs to comprehensively consider scanning efficiency and scanning accuracy. If the time interval is too short, the frequent movement of the robotic arm may lead to instability in the scanning process, affecting the accuracy of the scanning results; if the time interval is too long, it will reduce scanning efficiency. For example, it can move and scan once every 5 seconds to quickly find the first target area.

[0030] The preset scanning path defines the movement trajectory of the robotic arm within the tunnel. The design of the scanning path needs to consider the shape of the tunnel, the distribution of tunnel segments, and the possible locations of the target assembly points. For example, a spiral scanning path or a grid-like scanning path can be used to ensure that the robotic arm can cover the entire tunnel segment assembly surface.

[0031] When controlling the movement of the robotic arm, precise control of its speed and direction is required. The robotic arm's motion control system can generate corresponding control commands based on preset time intervals and scanning paths, driving the robotic arm to move to a designated position. Upon reaching the new position, the 3D scanning device scans again to acquire a new 3D scan image. Then, steps S111 and S112 are repeated to perform feature extraction and matching processing on the new image to determine if it contains the first target region. If it still does not contain it, the robotic arm continues to move and scan until an image containing the first target region is found.

[0032] Step S114: Determine the position of the first target region based on the current position of the robotic arm and the position of the first target region in the three-dimensional scan image.

[0033] Once the 3D scan image contains the first target region, it is necessary to combine the current position of the robotic arm with the position of the first target region in the image to determine the position of the first target region in actual space.

[0034] The current position of the robotic arm can be obtained through its own positioning system, which typically employs sensor technology such as laser rangefinders and encoders to measure the position and attitude of the robotic arm in real time. The position of the first target area in the 3D scanned image can be represented by an image coordinate system, usually described using pixel coordinates.

[0035] To convert image coordinates to actual spatial coordinates, a mapping relationship needs to be established between the image coordinate system and the actual spatial coordinate system. This process can be achieved through camera calibration. Camera calibration is the process of determining the camera's intrinsic parameters (such as focal length, principal point position, etc.) and extrinsic parameters (such as camera position and attitude, etc.). Through camera calibration, a transformation matrix between image coordinates and actual spatial coordinates can be obtained.

[0036] After obtaining the transformation matrix, the pixel coordinates of the first target region in the image are substituted into the transformation matrix to calculate its coordinates in actual space. This accurately determines the location of the first target region, providing precise positioning information for subsequent robotic arm movement and tunnel segment assembly operations.

[0037] Step S120: Based on the position of the first target area, move the end of the robotic arm to the visual image acquisition position corresponding to the first target area.

[0038] After determining the location of the first target area, the end effector of the robotic arm needs to be moved to the corresponding visual image acquisition position. The visual image acquisition position refers to the position where both planar and three-dimensional visual images of the first target area can be clearly and accurately acquired.

[0039] The robotic arm's motion control system calculates the distance and direction the end effector needs to move based on the position information of the first target area. Then, it generates corresponding control commands to drive the robotic arm to move along a predetermined trajectory to the visual image acquisition position. During the movement, the robotic arm's position and attitude need to be monitored in real time to ensure it accurately reaches the designated location.

[0040] For example, in a tunnel construction scenario, based on the location of the primary target area, the robotic arm's motion control system calculates the necessary movement of the arm's end effector in the X and Y directions, while simultaneously adjusting its pitch and yaw angles. Control commands are sent to the robotic arm's drive motors, which in turn move the arm's joints, gradually bringing the end effector closer to the visual image acquisition location. During this approach, the robotic arm's positioning system provides real-time feedback on its position, and the motion control system continuously adjusts the arm's movement based on this feedback, ultimately ensuring the end effector accurately reaches the visual image acquisition location.

[0041] Step S130: Acquire planar visual images and three-dimensional visual images of the first target area using a vision device deployed at the end of the robotic arm. The accuracy of the planar visual images and the three-dimensional visual images is higher than that of the three-dimensional scanned images. The installation position of the vision device at the end of the robotic arm is lower than that of the three-dimensional scanned image at the end of the robotic arm. The field of view of the vision device is smaller than that of the three-dimensional scanned image.

[0042] Once the end effector of the robotic arm reaches the visual image acquisition position, the vision device deployed at the end effector begins to work, acquiring planar and three-dimensional visual images of the first target area.

[0043] Vision devices may include, for example, planar vision cameras and 3D vision sensors. The planar vision camera is used to acquire planar visual images of the first target area, containing information such as the target area's color and texture. The 3D vision sensor is used to acquire 3D visual images of the first target area, which contain the target area's 3D spatial information, such as depth and shape.

[0044] Because the vision device is closer to the primary target area and has higher resolution and accuracy, the acquired planar and 3D visual images are more precise than the previous 3D scanned images. Simultaneously, the vision device is mounted lower at the end of the robotic arm than the 3D scanning device, resulting in a smaller field of view. This is because the primary task of the vision device is to perform high-precision local acquisition of the primary target area, while the 3D scanning device is used for large-scale scanning of the entire tunnel segment assembly surface.

[0045] For example, in the tunnel construction scenario described above, a planar vision camera and a 3D vision sensor begin to operate, acquiring planar and 3D visual images of the first target area, respectively. The image captured by the planar vision camera clearly shows the color and surface texture of the target assembly opening, while the image acquired by the 3D vision sensor accurately reflects the 3D shape and spatial position of the target assembly opening.

[0046] Step S140: Obtain the three-dimensional spatial data of the target assembly port based on the planar visual image and the three-dimensional visual image.

[0047] To achieve precise assembly of tunnel segments, it is necessary to acquire three-dimensional spatial data of the target assembly opening based on the collected planar and three-dimensional visual images. This process involves image processing and analysis in several steps.

[0048] Step S141: Filter the planar visual image to obtain a filtered planar visual image.

[0049] Planar vision images may be affected by noise during acquisition, such as Gaussian noise and salt-and-pepper noise, which can impact subsequent image processing and analysis. Therefore, it is necessary to filter planar vision images to remove noise and improve image quality.

[0050] Filtering can employ various algorithms, such as mean filtering, median filtering, and Gaussian filtering. In practical applications, the appropriate filtering algorithm can be selected based on the type of noise and the characteristics of the image. For example, if the image primarily contains Gaussian noise, Gaussian filtering can be chosen; if the image contains salt-and-pepper noise, median filtering can be selected.

[0051] For example, in a tunnel construction scenario, the acquired planar visual images contain some noise. To remove this noise, a Gaussian filtering algorithm is used. First, the kernel size and standard deviation of the Gaussian filter are determined. The kernel size determines the range of the neighborhood, while the standard deviation controls the shape of the Gaussian function. Then, for each pixel in the planar visual image, the weighted average of its neighboring pixels is calculated, and this average is used as the new value for the center pixel. After filtering, the noise in the image is effectively suppressed, and the image quality is improved.

[0052] Step S1411: Extract the illumination distribution features of the filtered planar visual image, wherein the illumination distribution features include the brightness value distribution of each pixel in the filtered planar visual image.

[0053] After filtering a planar visual image, it is necessary to extract its illumination distribution features to further improve image quality and the accuracy of subsequent processing. Illumination distribution features reflect the distribution of brightness values ​​among pixels in the image, and they have a significant impact on image segmentation, feature extraction, and recognition operations.

[0054] A brightness histogram can be obtained by iterating through each pixel in a filtered planar visual image, recording its brightness value, and counting the number of pixels with different brightness values. The brightness histogram visually displays the distribution of brightness values ​​in the image, and by analyzing the shape and characteristics of the histogram, the illumination conditions of the image can be understood.

[0055] For example, in a tunnel construction scenario, illumination distribution features are extracted from a filtered planar visual image. By traversing each pixel in the image, the brightness value of each pixel is obtained. Then, the number of pixels with different brightness values ​​is counted, and a brightness histogram is plotted. The histogram shows that there are some high-brightness and low-brightness areas in the image, indicating uneven illumination distribution.

[0056] Step S1412: Divide the filtered planar visual image into regions according to the illumination distribution characteristics to obtain a highlight region, a low-light region, and a normal illumination region, wherein the highlight region is a region with a brightness value higher than a first preset range, the low-light region is a region with a brightness value lower than a second preset range, and the normal illumination region is a region with a brightness value between the first preset range and the second preset range.

[0057] Based on the extracted illumination distribution features, the filtered planar visual image is divided into regions. The image is divided into highlight regions, low-light regions, and normal illumination regions to allow for targeted processing of different regions.

[0058] First, it is necessary to determine the first preset range and the second preset range. The first preset range is the brightness threshold range used to divide the highlight areas, and the second preset range is the brightness threshold range used to divide the low-light areas. These two ranges can be adjusted according to the actual situation to adapt to different image and lighting conditions.

[0059] Then, each pixel in the filtered planar visual image is traversed, and the pixel is divided into the corresponding region based on the comparison result of its brightness value with the first preset range and the second preset range. If the brightness value of a pixel is higher than the first preset range, it is divided into a highlight region; if the brightness value of a pixel is lower than the second preset range, it is divided into a low-light region; if the brightness value of a pixel is between the first preset range and the second preset range, it is divided into a normal lighting region.

[0060] For example, in the tunnel construction scenario described above, based on the characteristics of light distribution, a first preset range is defined as a brightness value greater than threshold A, and a second preset range is defined as a brightness value less than threshold B. The filtered planar visual image is then iterated through, with pixels having a brightness value greater than threshold A marked as highlight areas, pixels having a brightness value less than threshold B marked as low-light areas, and pixels with brightness values ​​between threshold A and threshold B marked as normal lighting areas.

[0061] Step S1413: Perform adaptive gamma compression processing on the highlight area. The adaptive gamma compression processing includes determining gamma compression parameters based on the average brightness value of the highlight area and using the gamma compression parameters to reduce the brightness of overexposed pixels in the highlight area.

[0062] For the identified highlight areas, their excessively high brightness values ​​may lead to a loss of detail in the image. To restore the details of the highlight areas, adaptive gamma compression is required.

[0063] Specifically, gamma compression parameters can be determined based on the average brightness value of the highlight area. These parameters control the intensity of the gamma transform, with different average brightness values ​​corresponding to different gamma compression parameters. A mapping relationship between average brightness values ​​and gamma compression parameters can be established to find the corresponding gamma compression parameter based on the average brightness value of the highlight area.

[0064] After determining the gamma compression parameters, a gamma transform is performed on each pixel within the highlight region. The formula for the gamma transform is: Output pixel value = Input pixel value raised to the power of gamma. Through gamma transform, the brightness of overexposed pixels in the highlight region can be reduced while preserving image details.

[0065] For example, in the tunnel construction scenario described above, the average brightness value of the highlight area is calculated, and the corresponding gamma compression parameters are found based on a pre-established mapping relationship. Then, a gamma transform is performed on each pixel within the highlight area to reduce the brightness of overexposed pixels. After gamma compression processing, the brightness of the highlight area is effectively controlled, and the details in the image are restored.

[0066] Step S1414: Perform adaptive histogram equalization processing on the low-light area. The adaptive histogram equalization processing includes determining the range of histogram equalization based on the pixel brightness distribution of the low-light area, and using the range to improve the contrast of dark details in the low-light area.

[0067] In low-light areas, due to their low brightness values, details in the shadows of the image may not be clearly visible. To improve the contrast of shadow details in low-light areas, adaptive histogram equalization is required.

[0068] Specifically, the range of histogram equalization can be determined based on the pixel brightness distribution in low-light areas. The range of histogram equalization determines which brightness values ​​of pixels will be adjusted. By analyzing the brightness histogram of low-light areas, the distribution range of brightness values ​​can be determined, and then the start and end points of histogram equalization can be determined based on this range.

[0069] After determining the range for histogram equalization, histogram equalization is performed on pixels in low-light regions. Histogram equalization enhances image contrast by adjusting the brightness distribution of the image. It redistributes the image's brightness histogram, making the brightness more uniform and thus improving the contrast of details in dark areas.

[0070] By remapping the brightness values ​​of pixels in low-light areas, the pixel distribution that was originally concentrated in the low-light range is made more dispersed, thereby enhancing the visibility of details in dark areas.

[0071] Specifically, we can first count the number of pixels corresponding to each brightness value in the low-light region to form a brightness histogram. Then, based on the determined histogram equalization range, we calculate the new brightness value corresponding to each brightness value after equalization. This calculation process is based on the principle of the cumulative distribution function, mapping the original brightness values ​​to a new brightness range according to certain rules.

[0072] For example, assuming the original low-light region's brightness values ​​range from brightness value a to brightness value b, and the histogram equalization range determined through analysis is from brightness value c to brightness value d, then for a pixel with a brightness value e within the low-light region, its new brightness value f in the new brightness range (brightness value c to brightness value d) will be calculated based on the cumulative distribution function. By processing all pixels within the low-light region in this way, the contrast of the low-light region is significantly improved, and previously difficult-to-see details in dark areas become clearer.

[0073] Step S1415: The processed highlight area, low light area and normal lighting area are merged at the pixel level to obtain an enhanced planar visual image.

[0074] After processing the highlight and low-light areas separately, the processed highlight and low-light areas, along with the unprocessed normal-light areas, need to be merged pixel-by-pixel to obtain an enhanced planar visual image. Pixel-by-pixel merging means that the pixels of each region are accurately combined according to their positions in the original image to form a complete image.

[0075] Specifically, each pixel position in the original planar visual image can be traversed to determine which region the pixel belongs to (highlight region, lowlight region, or normal lighting region). Then, the pixel value processed for the corresponding region is placed at that position. For example, for a given pixel position, if it was marked as a highlight region during region segmentation, the pixel value at that position, after adaptive gamma compression, is placed at the corresponding position in the merged image; if it was marked as a lowlight region, the pixel value after adaptive histogram equalization is placed; if it is a normal lighting region, the original pixel value is directly placed. In this way, pixel-level merging of the three regions is finally completed, resulting in a more uniformly lit and more detailed enhanced planar visual image.

[0076] Step S142: Based on the target detection threshold determined by using the edge of the tunnel segment joint as the threshold dividing line, perform region segmentation processing on the filtered planar visual image, remove invalid data regions in the filtered planar visual image, and retain the second target region in the filtered planar visual image that includes the target assembly port.

[0077] To accurately extract the region containing the target assembly joint from the filtered planar visual image, region segmentation is required. Here, the target detection threshold is determined based on the edge of the tunnel segment joint. First, the edge of the tunnel segment joint exhibits significant feature changes in the image, typically manifested as a sharp change in pixel grayscale values. Analyzing these feature changes provides crucial references for region segmentation.

[0078] Optionally, if the image has undergone illumination enhancement processing, then region segmentation processing is performed on the enhanced planar visual image to remove invalid data regions in the filtered planar visual image and retain the second target region in the filtered planar visual image that includes the target assembly port.

[0079] The following section uses the example of performing region segmentation on the filtered planar visual image to illustrate this process.

[0080] Step S1421: Extract the gradient features of the tunnel segment joint edges in the filtered planar visual image, wherein the gradient features include the grayscale change rate of pixels at the tunnel segment joint edges.

[0081] To accurately determine the edges of tunnel segment joints, it is necessary to extract their gradient features. Gradient features reflect the rate of change of pixel grayscale values ​​in an image. At the edges of tunnel segment joints, pixel grayscale values ​​change significantly, resulting in larger gradient values. For example, methods such as the Sobel operator can be used to calculate the gradient of each pixel in the image.

[0082] The Sobel operator can perform convolution operations on the image in both the horizontal and vertical directions to obtain the gradient components in the horizontal and vertical directions respectively. These two components are then combined using Euclidean distance to obtain the gradient value of each pixel.

[0083] For pixels on the edge of tunnel segment joints, their gradient values ​​will be significantly higher than those of pixels in other areas. For example, the gradient values ​​of pixels along the edge of a tunnel segment joint in an image may form a distinct high-value band. By detecting this high-value band, the approximate location of the tunnel segment joint edge can be determined.

[0084] Step S1422: Calculate the mean and variance of the gradient features of the tunnel segment joint edge. The mean is the average of the gradient values ​​of all pixels at the tunnel segment joint edge, and the variance is the average of the sum of squared deviations of the gradient values ​​of all pixels at the tunnel segment joint edge from the mean.

[0085] After extracting the gradient features of the tunnel segment joint edges, the mean and variance of these gradient features are calculated. The mean reflects the overall level of the gradient values ​​at the tunnel segment joint edges, while the variance reflects the degree of dispersion of the gradient values.

[0086] To calculate the mean, the gradient values ​​of all pixels at the edge of the tunnel segment joint can be summed and then divided by the number of pixels to obtain the average gradient value. To calculate the variance, the deviation of the gradient value of each pixel from the mean can be calculated first, these deviations can be squared and summed, and then divided by the number of pixels to obtain the variance value.

[0087] For example, if there are multiple pixels on the edge of a tunnel segment joint, with gradient values ​​g1, g2, g3, etc., these gradient values ​​are first summed and then divided by the number of pixels to obtain the mean. Then, for each gradient value, the deviation from the mean is calculated, the squares of the deviations are summed, and finally divided by the number of pixels to obtain the variance. These two statistical measures, mean and variance, help to understand the distribution of gradient features at the edge of the tunnel segment joint, providing important basis for subsequent adjustment of the target detection threshold.

[0088] Step S1423: Dynamically adjust the target detection threshold according to the mean and variance of the gradient features. If the variance of the gradient features is greater than or equal to a preset variance value, the target detection threshold is reduced to retain more edge details. If the variance of the gradient features is less than the preset variance value, the target detection threshold is increased to remove noise.

[0089] The target detection threshold can be dynamically adjusted based on the mean and variance of the gradient features at the edges of the tunnel segment joints obtained from the calculation.

[0090] The preset variance value is a pre-defined reference value used to determine whether the dispersion of gradient values ​​is within a reasonable range. When the variance of the gradient feature is greater than or equal to the preset variance value, the gradient value distribution representing the edge of the tunnel segment joint is relatively scattered, and some edge details may not be fully detected. In this case, lowering the target detection threshold allows more pixels that were originally ignored to be included in the edge detection range, thereby preserving more edge details.

[0091] Conversely, when the variance of the gradient features is less than a preset variance value, the gradient values ​​are relatively concentrated, which may lead to noise interference and inaccurate edge detection. In this case, increasing the target detection threshold excludes some pixels with low gradient values, thereby removing noise and making edge detection more accurate.

[0092] For example, if the preset variance value is the variance h, and the calculated variance of the gradient feature of the tunnel segment joint edge is the variance i, when the variance i is greater than or equal to the variance h, the target detection threshold is reduced from the threshold j to the threshold k; when the variance i is less than the variance h, the target detection threshold is increased from the threshold j to the threshold l.

[0093] Step S1424: Select multiple sampling points on the edge of the tunnel segment joint, wherein the multiple sampling points are evenly distributed at different positions on the edge of the tunnel segment joint.

[0094] To verify the effectiveness of the adjusted target detection threshold, multiple sampling points can be selected at the edge of the tunnel segment joint. These sampling points are evenly distributed at different locations on the edge of the tunnel segment joint, which can comprehensively reflect the characteristics of the tunnel segment joint edge.

[0095] For example, sampling can be performed on the edges of tunnel segment joints according to preset rules, such as selecting a sampling point at certain pixel intervals. By detecting and analyzing these sampling points, it is possible to better determine whether the adjusted target detection threshold can accurately segment the edges of tunnel segment joints.

[0096] Step S1425: Check whether the edges after threshold segmentation at each sampling point are continuous, and whether the edges between adjacent sampling points are continuous, in order to verify the continuity of the edges after the adjusted target detection threshold segmentation.

[0097] After selecting sampling points, check whether the edges at each sampling point are continuous after threshold segmentation, and whether the edges between adjacent sampling points are continuous.

[0098] Edge continuity is a crucial indicator for evaluating region segmentation performance. Discontinuous edges can lead to inaccurate extraction of the target assembly area. For each sampling point, edge continuity can be determined by examining the gradient values ​​of surrounding pixels and the threshold segmentation results. For edge continuity between adjacent sampling points, the pixel connectivity between the two points can be compared. For example, if there are obvious breaks between adjacent sampling points, indicating insufficient edge continuity, further adjustment of the target detection threshold may be necessary.

[0099] Step S1426: If the edge continuity is insufficient, the target detection threshold is fine-tuned again. The fine-tuning includes adjusting the size of the target detection threshold according to the degree of insufficient continuity.

[0100] If insufficient edge continuity is found during the inspection process, the target detection threshold needs to be fine-tuned again. This fine-tuning involves adjusting the target detection threshold based on the degree of continuity deficiency.

[0101] If the edge breakage is severe, indicating that the target detection threshold is set too high or too low, a significant adjustment to the threshold is needed. If the edge only has slight discontinuities, the threshold can be adjusted slightly. For example, if a large break is found between adjacent sampling points, the target detection threshold may be set too high, excluding some pixels that should belong to the edge. In this case, the target detection threshold can be appropriately lowered. If there are some small gaps at the edge, the threshold may be slightly too high, and it can be slightly lowered to make the edge more continuous.

[0102] The target detection threshold is continuously fine-tuned until the edge continuity meets the requirements.

[0103] Step S1427: Perform the region segmentation process on the filtered planar visual image based on the final adjusted target detection threshold, remove invalid data regions, and retain the second target region containing the target assembly port.

[0104] After going through the above series of steps, a suitable target detection threshold is finally determined, and this threshold is used to perform region segmentation processing on the filtered planar visual image.

[0105] Specifically, pixels in the image can be classified based on a comparison of their gradient values ​​with a target detection threshold. Pixels with gradient values ​​greater than or equal to the target detection threshold are considered to belong to edge or target regions, while pixels with gradient values ​​less than the target detection threshold are considered to be invalid data regions. In this way, invalid data regions are eliminated, and the second target region containing the target assembly port is retained.

[0106] For example, for each pixel in a planar visual image, its gradient value is calculated. If the gradient value is greater than or equal to the final adjusted target detection threshold, the pixel is marked as belonging to the second target region; if the gradient value is less than the target detection threshold, the pixel is marked as an invalid data region. Finally, the pixels belonging to the second target region are retained to form an image containing only the target assembly port and related areas.

[0107] Step S143: Obtain the target assembly port feature vector dataset generated by the pre-trained deep learning model.

[0108] To accurately identify the target assembly port from the second target region, it is necessary to obtain a dataset of target assembly port feature vectors generated by a pre-trained deep learning model. This deep learning model is trained on a large amount of target assembly port sample data and is able to learn the feature patterns of the target assembly ports.

[0109] Deep learning models can employ a Convolutional Neural Network (CNN) architecture. A CNN typically consists of multiple convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract local features from an image by sliding convolution kernels across the image to generate feature maps. Pooling layers downsample the feature maps, reducing the amount of data while preserving important feature information. Fully connected layers convert the feature maps into feature vectors for tasks such as classification or regression.

[0110] When training a deep learning model, a large amount of target assembly port sample data needs to be prepared, and each sample data needs to be labeled accordingly. This sample data can come from different tunnel scenarios and different tunnel segment types to improve the model's generalization ability. During training, optimization algorithms (such as stochastic gradient descent) can be used to adjust the model's parameters so that the model's output is as close as possible to the labeled results. Specifically, the sample data is input into the model, and the prediction result is obtained through forward propagation. Then, the loss function value between the prediction result and the label is calculated. Based on the loss function value, the optimization algorithm is used to update the model's parameters, gradually reducing the loss function value. After multiple iterations of training, the model's performance gradually improves, enabling it to accurately learn the features of the target assembly port.

[0111] After the deep learning model is trained, each target assembly port sample from the training dataset is input into the model. Through forward propagation, the corresponding feature vector is extracted from the fully connected layers or specific layers of the model. These feature vectors form the target assembly port feature vector dataset. Each feature vector is a multi-dimensional vector containing the feature information of the target assembly port. For example, for a target assembly port sample, after being input into the model and processed by intermediate layers, the feature vector of the target assembly port sample is output from the fully connected layer. Each dimension of this feature vector represents the numerical value of the target assembly port in a certain feature. Collecting the feature vectors corresponding to all training samples forms the target assembly port feature vector dataset.

[0112] For example, the target assembly port feature vector dataset may include edge contour features, geometric shape features, and surface texture features. Edge contour features can reflect the shape boundary information of the target assembly port and are used to describe the physical form of the target assembly port. Geometric shape features can describe the shape attributes of the target assembly port as a whole, such as the length-to-width ratio and diagonal length. Surface texture features reflect the microstructure and details of the target assembly port surface, such as the surface roughness and texture direction.

[0113] Step S144: Perform feature data matching processing on the target assembly port feature vector dataset and the second target region, and filter out the two-dimensional contour feature point pixel information of the target assembly port from the second target region according to the matching confidence of the feature data matching processing.

[0114] After obtaining the target assembly port feature vector dataset and the second target region, feature data matching processing is performed. By comparing the features in the second target region with the features in the target assembly port feature vector dataset, the parts with high matching degrees are found.

[0115] First, feature vectors are extracted from the second target region. This can be done using the same method as when training a deep learning model: the second target region is input into the model, and after forward propagation, the corresponding feature vectors are extracted from specific layers of the deep learning model. For example, the image data of the second target region is input into a trained CNN model, processed through convolutional layers, pooling layers, and fully connected layers, with the fully connected layer outputting a feature vector. This feature vector represents the feature information of the second target region.

[0116] Then, the extracted feature vectors of the second target region are compared with each feature vector in the target assembly port feature vector dataset. For example, similarity calculation methods (such as cosine similarity, Euclidean distance, etc.) can be used to measure the similarity between two feature vectors. For each target assembly port feature vector, its similarity score with the feature vectors of the second target region is calculated. For example, when using cosine similarity calculation, the two feature vectors are dot-producted and then divided by the product of their magnitudes to obtain the cosine similarity score. The higher the score, the more similar the two feature vectors are, meaning a higher degree of feature matching between a portion of the second target region and the target assembly port.

[0117] Next, the matching confidence score is calculated based on the similarity score of the feature matching. The matching confidence score reflects the reliability of the match between a portion of the second target region and the target assembly port. The matching confidence score can be calculated by normalizing the similarity score or using other statistical analysis methods. For example, all similarity scores can be sorted, and a threshold can be determined based on the score distribution. Scores above the threshold are considered to have a high degree of matching and correspondingly high matching confidence; scores below the threshold are considered to have a low degree of matching and correspondingly low matching confidence.

[0118] Finally, based on the matching confidence score, the two-dimensional contour feature point pixel information of the target assembly port is filtered out from the second target region. For regions with high matching confidence scores, the corresponding pixel positions are found in the second target region; these pixel positions represent the two-dimensional contour feature point pixel information of the target assembly port. The two-dimensional contour feature points can be determined by marking the boundary pixels of the matching region or using other edge detection methods. For example, for regions with high matching confidence scores, edge detection operators (such as the Canny operator) are used to find the edge pixels of that region; the coordinates of these edge pixels represent the two-dimensional contour feature point pixel information.

[0119] Step S145: Based on the data mapping relationship between the planar visual image and the three-dimensional visual image, perform a three-dimensional spatial coordinate mapping transformation on the pixel information of the two-dimensional contour feature points to obtain the three-dimensional spatial data of the target assembly port.

[0120] In order to convert the two-dimensional contour feature point pixel information obtained from the planar visual image into the three-dimensional spatial data of the target assembly port, it is necessary to perform three-dimensional spatial coordinate mapping transformation using the data mapping relationship between the planar visual image and the three-dimensional visual image.

[0121] Step S1451: Perform preprocessing operations on the three-dimensional visual image to obtain a preprocessed three-dimensional visual image. The preprocessing operations include at least one of the following: noise reduction processing, image enhancement processing, and downsampling processing.

[0122] Preprocessing 3D visual images can improve image quality and provide more accurate data for subsequent mapping transformations. Noise reduction can be achieved using filtering algorithms (such as Gaussian filtering and median filtering) to eliminate noise points in the image. For example, when using Gaussian filtering, a weighted average of each pixel and its neighboring pixels is applied according to a Gaussian function to remove random noise.

[0123] Image enhancement processing can employ methods such as histogram equalization to enhance the contrast and brightness of an image, making its features more prominent.

[0124] Downsampling reduces the amount of data by decreasing the number of pixels in an image while preserving important feature information. For example, 3D visual images can be downsampled by alternating row and column sampling. In practice, one or more preprocessing operations can be selected depending on the specific characteristics of the 3D visual image.

[0125] Step S1452: Based on the difference in field of view deflection between the preprocessed 3D visual image and the planar visual image, the preprocessed 3D visual image is projected to obtain a projected 3D visual image. The difference in field of view deflection between the projected 3D visual image and the planar visual image is less than or equal to a preset difference threshold.

[0126] Because the acquisition perspectives of planar and 3D visual images may differ, their fields of view may be skewed. To make their fields of view more consistent, projection processing is required.

[0127] Step S14521: Extract the target feature point set from the preprocessed 3D visual image. The target feature point set includes edge feature points of tunnel segments and splicing seam feature points. The edge feature points include continuous points on the outline of the tunnel segments, and the splicing seam feature points include the endpoints and midpoints of the gaps between adjacent tunnel segments.

[0128] Extract the target feature point set from the preprocessed 3D visual image. Edge detection algorithms (such as the Canny operator) can be used to extract the edge feature points of the tunnel segments, which constitute continuous points on the outline of the tunnel segments.

[0129] For splicing seam feature points, the endpoints and midpoints of the gaps between adjacent tunnel segments can be identified by analyzing the gaps. For example, for a gap between adjacent tunnel segments, the gradient values ​​of pixels on both sides of the gap are calculated, and the locations with larger gradient value changes are identified as the endpoints of the gap; then, the midpoint of the gap is found using a midpoint calculation method. These edge feature points and splicing seam feature points constitute the target feature point set.

[0130] Step S14522: Extract the target feature point set from the planar visual image. The target feature point set corresponds to the same tunnel segment edge and splicing seam as the target feature point set in the preprocessed three-dimensional visual image.

[0131] The method for extracting the set of target feature points in a planar visual image is similar to that for extracting them in a preprocessed 3D visual image. Similarly, edge detection algorithms can be used to find the edge feature points of tunnel segments; these edge feature points are continuous points on the outline of the tunnel segments in the planar visual image.

[0132] For splice seam feature points, the endpoints and midpoints of the gaps can also be determined by analyzing the gaps between adjacent tunnel segments. In this step, it is essential to ensure that the extracted set of target feature points corresponds to the same tunnel segment edges and splice seams as the set of target feature points in the preprocessed 3D visual image, in order to accurately compare the differences in field of view deflection between the two.

[0133] For example, in a planar visual image, the location with the largest change in gray value can be identified by calculating the change in pixel gray value between adjacent tunnel segments as the endpoint of the splicing seam, and the midpoint between the endpoints can be calculated as the midpoint of the splicing seam, thus obtaining a set of target feature points at the same location as those in the three-dimensional visual image.

[0134] Step S14523: Compare the relative positional relationship between the target feature point set in the preprocessed 3D visual image and the target feature point set in the planar visual image to obtain descriptive information of the field of view deflection difference. The descriptive information includes the offset of the target feature point in the horizontal direction, the offset of the target feature point in the vertical direction, and the rotation angle of the target feature point.

[0135] The set of target feature points in the preprocessed 3D visual image is compared with the set of target feature points in the planar visual image. For each pair of corresponding target feature points (such as the endpoint of a stitching seam in the 3D visual image and the corresponding endpoint of a stitching seam in the planar visual image), the coordinate differences between them in the horizontal and vertical directions are calculated. These differences are the offsets of the target feature points in the horizontal and vertical directions.

[0136] Simultaneously, by analyzing the relative rotation relationships between target feature points, the rotation angle of the target feature points can be obtained. For example, for a pair of corresponding edge feature points, the rotation angle can be determined by calculating the change in the angle between the line connecting them and the horizontal direction. By performing such analysis and calculation on all corresponding target feature points, descriptive information about the overall field-of-view deflection difference can be obtained, including the horizontal offset, vertical offset, and rotation angle.

[0137] Step S14524: Based on the description information of the field of view deflection difference, generate projection adjustment rules, which include a first parameter for correcting horizontal offset, a second parameter for correcting vertical offset, and a third parameter for correcting rotation angle.

[0138] Based on the descriptive information of the obtained field-of-view deflection differences, projection adjustment rules are generated. For the horizontal offset, it is used as the first parameter to correct the horizontal offset; for the vertical offset, it is used as the second parameter to correct the vertical offset. For the rotation angle, it is converted into a suitable parameter form as the third parameter to correct the rotation angle.

[0139] For example, if the horizontal offset is offset m, then the first parameter is set to a value related to offset m, which is used to adjust the horizontal coordinates in the subsequent projection transformation; similarly, the vertical offset is offset n, the second parameter is related to offset n; the rotation angle is angle p, and the third parameter is set according to angle p, which is used to rotate the target feature point in the projection transformation.

[0140] Step S14525: Apply the projection adjustment rules to perform a projection transformation operation on the preprocessed 3D visual image to obtain the projected 3D visual image. The projection transformation operation includes adjusting the horizontal coordinates of each point in the 3D visual image according to the first parameter, adjusting the vertical coordinates of each point in the 3D visual image according to the second parameter, and rotating each point in the 3D visual image according to the third parameter.

[0141] The generated projection adjustment rules are applied to perform a projection transformation operation on the preprocessed 3D visual image. For each point in the 3D visual image, its horizontal coordinate is adjusted according to a first parameter; if the first parameter is positive, the horizontal coordinate of the point is increased accordingly; if it is negative, it is decreased accordingly. Similarly, the vertical coordinate of the point is adjusted according to a second parameter. For rotation operations, the point is rotated according to a third parameter. For example, using a center point of the image as the rotation center, the coordinate position after rotation is calculated according to the third parameter. By performing such operations on all points in the 3D visual image, a projected 3D visual image can be obtained. The difference in field of view deflection between this projected 3D visual image and the planar visual image is less than or equal to a preset difference threshold.

[0142] Step S1453: Based on the pixel coordinate scale of the planar visual image, perform data normalization processing on the projected 3D visual image to obtain a normalized 3D visual image.

[0143] In order to enable better data mapping between planar visual images and projected 3D visual images, it is necessary to perform data normalization on the projected 3D visual images so that their coordinate scale is consistent with the pixel coordinate scale of the planar visual images.

[0144] Step S14531: Extract the pixel coordinate scale information of the planar visual image. The pixel coordinate scale information includes the horizontal spacing and vertical spacing of the pixels. The horizontal spacing is the distance between two adjacent pixels in the horizontal direction, and the vertical spacing is the distance between two adjacent pixels in the vertical direction.

[0145] When extracting pixel coordinate scale information from a planar visual image, it can be determined by analyzing the imaging principle and related parameters of the planar visual image. For example, based on the camera's resolution and the physical size of the imaging sensor, the actual distances between two adjacent pixels in the horizontal and vertical directions can be calculated, serving as the horizontal and vertical pixel spacing, respectively. Assuming the camera's horizontal resolution is resolution x and the imaging sensor's physical size in the horizontal direction is size y, the horizontal pixel spacing can be obtained by dividing size y by resolution x. Similarly, the vertical pixel spacing can be calculated.

[0146] Step S14532: Extract the coordinate scale information of the three-dimensional visual image after projection processing. The coordinate scale information includes the horizontal unit length and the vertical unit length of the three-dimensional coordinates. The horizontal unit length is the actual distance between two adjacent coordinate points in the horizontal direction in the three-dimensional visual image, and the vertical unit length is the actual distance between two adjacent coordinate points in the vertical direction in the three-dimensional visual image.

[0147] For a projected 3D visual image, its coordinate scale information can be determined through the parameters and scanning principle of the 3D scanning device. When acquiring data, the 3D scanning device records the 3D coordinates of each point. By analyzing the distance between adjacent points, the horizontal and vertical unit lengths of the 3D coordinates can be obtained. For example, by comparing the difference in horizontal coordinates between two adjacent points, the horizontal unit length is obtained; by comparing the difference in vertical coordinates between two adjacent points, the vertical unit length is obtained.

[0148] Step S14533: Compare the pixel coordinate scale information of the planar visual image with the coordinate scale information of the three-dimensional visual image after projection processing to obtain scale difference description information, which includes the proportional relationship of the horizontal scale and the proportional relationship of the vertical scale.

[0149] The pixel coordinate scale information of the planar visual image is compared with the coordinate scale information of the projected 3D visual image. The ratio of the horizontal spacing of pixels in the planar visual image to the horizontal unit length of the 3D coordinates in the 3D visual image is calculated to obtain the horizontal scale ratio. The ratio of the vertical spacing of pixels in the planar visual image to the vertical unit length of the 3D coordinates in the 3D visual image is calculated to obtain the vertical scale ratio.

[0150] For example, if the horizontal spacing of pixels in a planar visual image is spacing *a*, and the horizontal unit length of the three-dimensional coordinates in a three-dimensional visual image is length *b*, then the ratio of the horizontal scale is the ratio of spacing *a* to length *b*. Similarly, the ratio of the vertical scale can be obtained.

[0151] Step S14534: Generate scale adjustment rules based on the scale difference description information. The scale adjustment rules include a horizontal scale parameter for adjusting the horizontal scale and a vertical scale parameter for adjusting the vertical scale.

[0152] Based on the obtained scale difference description information, scale adjustment rules are generated. The horizontal scale ratio is used as the horizontal scale parameter for adjusting the horizontal scale, and the vertical scale ratio is used as the vertical scale parameter for adjusting the vertical scale. For example, if the horizontal scale ratio is ratio c, then the horizontal scale parameter is set to ratio c; if the vertical scale ratio is ratio d, then the vertical scale parameter is set to ratio d.

[0153] Step S14535: Apply the scale adjustment rule to adjust the coordinate scale of the projected three-dimensional visual image. The adjustment operation includes scaling the horizontal coordinate values ​​of each point in the three-dimensional visual image according to the scale parameter of the horizontal scale, and scaling the vertical coordinate values ​​of each point in the three-dimensional visual image according to the scale parameter of the vertical scale.

[0154] The generated scaling rules are applied to adjust the coordinate scale of the projected 3D visual image. For each point in the 3D visual image, its horizontal coordinate value is multiplied by the horizontal scaling parameter to achieve horizontal scaling; its vertical coordinate value is multiplied by the vertical scaling parameter to achieve vertical scaling.

[0155] For example, if the horizontal coordinate of a point is value e and the horizontal scale parameter is scale c, then the adjusted horizontal coordinate is value e multiplied by scale c. Similarly, the vertical coordinate values ​​are adjusted accordingly.

[0156] Step S14536: Verify whether the coordinate scale of the adjusted 3D visual image is consistent with the pixel coordinate scale of the planar visual image.

[0157] To verify whether the coordinate scale of the adjusted 3D visual image is consistent with the pixel coordinate scale of the planar visual image, the horizontal and vertical spacing can be compared again. The horizontal and vertical distances between adjacent coordinate points in the adjusted 3D visual image are recalculated and compared with the pixel horizontal and vertical spacing of the planar visual image. For example, the new horizontal distance between two adjacent coordinate points in the adjusted 3D visual image is calculated and compared with the pixel horizontal spacing of the planar visual image. If the difference is within the allowable error range, the horizontal scale is considered consistent; similarly, the consistency of the vertical scale is verified.

[0158] Step S14537: If they are consistent, the 3D visual image after the adjustment operation is determined as the 3D visual image after the normalization process; if the scales are inconsistent, the horizontal scale parameter and / or the vertical scale parameter in the scale adjustment rule are adjusted, and the operation of applying the scale adjustment rule and verifying the scale is repeated until the scales are consistent.

[0159] If the verification results show that the coordinate scale of the adjusted 3D visual image is consistent with the pixel coordinate scale of the planar visual image, then the adjusted 3D visual image is determined to be the normalized 3D visual image. If the scales are inconsistent, the horizontal and / or vertical scaling parameters in the scale adjustment rules need to be adjusted. For example, if the horizontal scales are still inconsistent, the horizontal scaling parameter can be fine-tuned; if the vertical scales are inconsistent, the vertical scaling parameter can be fine-tuned. Then, the adjusted scale adjustment rules are applied again to adjust the coordinate scale of the 3D visual image, and the scale consistency is verified again. This process is repeated until the scales of the two images are consistent.

[0160] Step S1454: Based on the data mapping relationship between the planar visual image and the normalized three-dimensional visual image, perform a three-dimensional spatial coordinate mapping transformation on the pixel information of the two-dimensional contour feature points to obtain the three-dimensional spatial data of the target assembly port.

[0161] After obtaining the normalized 3D visual image, the 3D spatial coordinate mapping transformation of the pixel information of the 2D contour feature points is performed by utilizing the data mapping relationship between the 2D visual image and the normalized 3D visual image.

[0162] Step S14541: Obtain the attitude sensor data of the end effector of the robotic arm. The attitude sensor data includes the position coordinates, pitch angle, yaw angle, and roll angle of the end effector of the robotic arm.

[0163] In this embodiment, an attitude sensor is installed at the end effector of the robotic arm to acquire its attitude information in real time. The attitude sensor can be implemented using various technologies, such as an inertial measurement unit (IMU). The attitude sensor data includes the position coordinates of the robotic arm's end effector, representing its specific position in three-dimensional space; the pitch angle, representing the rotation angle of the end effector about the horizontal axis; the yaw angle, representing the rotation angle about the vertical axis; and the roll angle, representing the rotation angle about the robotic arm's own axis. This attitude sensor data is crucial for accurate data mapping and coordinate transformation.

[0164] Step S14542: Calculate the acquisition view offset of the vision device based on the attitude sensor data. The acquisition view offset includes the tilt angle of the vision device relative to the preset acquisition view.

[0165] Based on the attitude sensor data acquired at the end of the robotic arm, the viewing angle offset of the vision device can be calculated. Since the vision device is mounted at the end of the robotic arm, changes in the arm's attitude will cause a shift in the viewing angle of the vision device. By analyzing and calculating the attitude sensor data, the tilt angle of the vision device relative to the preset viewing angle can be obtained. For example, based on the values ​​of pitch, yaw, and roll angles, the tilt angles of the vision device in various directions can be calculated; these tilt angles are the viewing angle offset.

[0166] Step S14543: Integrate the acquisition viewpoint offset into the mapping matrix between the planar visual image and the normalized 3D visual image to obtain a corrected mapping matrix. The integration includes converting the acquisition viewpoint offset into a rotation matrix and multiplying the rotation matrix with the original mapping matrix.

[0167] To account for the impact of the viewing angle offset on data mapping, the viewing angle offset needs to be incorporated into the mapping matrix between the planar visual image and the normalized 3D visual image. First, the viewing angle offset is converted into a rotation matrix. The rotation matrix is ​​a matrix representing a rotation transformation, calculated based on the tilt angle of the viewing angle offset. Then, the rotation matrix is ​​multiplied by the original mapping matrix to obtain the corrected mapping matrix. For example, if the original mapping matrix is ​​matrix A and the rotation matrix is ​​matrix B, then the corrected mapping matrix is ​​the product of matrix A and matrix B.

[0168] Step S14544: Based on the corrected mapping matrix, perform a three-dimensional spatial coordinate mapping transformation on the pixel information of the two-dimensional contour feature points to obtain the three-dimensional spatial data of the target assembly port. The transformation includes substituting the pixel coordinates of the pixel information of the two-dimensional contour feature points into the corrected mapping matrix to obtain the corresponding three-dimensional spatial coordinates.

[0169] The pixel information of the two-dimensional contour feature points is transformed into three-dimensional spatial coordinates based on the modified mapping matrix. The pixel coordinates of the two-dimensional contour feature points are substituted into the modified mapping matrix, and the corresponding three-dimensional spatial coordinates are obtained through matrix operations.

[0170] For example, for a pixel coordinate (coordinate value f, coordinate value g) in the pixel information of a two-dimensional contour feature point, it is substituted into the corrected mapping matrix, and after matrix multiplication and addition, the corresponding three-dimensional spatial coordinate (coordinate value h, coordinate value i, coordinate value j) is obtained. By performing the above operations on all pixel coordinates in the pixel information of the two-dimensional contour feature point, the three-dimensional spatial data of the target assembly port is obtained. The three-dimensional spatial data is used to describe the position and shape of the target assembly port in three-dimensional space.

[0171] Step S150: Based on the three-dimensional spatial data of the target assembly port, control the robotic arm to perform the assembly operation of the tunnel segment to be assembled.

[0172] After obtaining the three-dimensional spatial data of the target assembly port, the robotic arm's control system can use this data to control the robotic arm to perform the assembly operation of the tunnel segments to be assembled.

[0173] Specifically, the control system of the robotic arm can use the three-dimensional spatial data of the target assembly port as the target position and attitude information, and combine it with the robotic arm's own kinematic and dynamic models to calculate the motion parameters of each joint of the robotic arm.

[0174] For example, based on the position and orientation of the target assembly port, the required rotation angle and movement distance of each joint of the robotic arm are calculated. Then, the control system sends control commands to the drive unit of the robotic arm, which drives the movement of each joint of the robotic arm, so that the end of the robotic arm carrying the tunnel segment to be assembled moves accurately to the position of the target assembly port and assembles it in the correct orientation.

[0175] During assembly, the robotic arm can also monitor the contact and positional deviation between the tunnel segment and the target assembly opening in real time using sensors. Based on the monitoring results, the robotic arm's movement can be adjusted promptly to ensure the accuracy and stability of the tunnel segment assembly. For example, if the sensors detect a certain positional deviation between the tunnel segment and the target assembly opening, the robotic arm can fine-tune its position and attitude to accurately embed the tunnel segment into the target assembly opening, completing the positioning and installation task of the tunnel segment.

[0176] The method provided in this application embodiment comprehensively utilizes 3D scanning equipment and vision equipment. First, the 3D scanning equipment is used to perform large-scale preliminary positioning to determine the approximate location of the first target area containing the target assembly port. Then, with the help of a vision device installed at a lower position at the end of the robotic arm, which has a smaller field of view but higher precision, planar and 3D visual images of the first target area are acquired, thereby achieving more accurate local information acquisition of the target assembly port. In the image processing, the planar visual image undergoes multiple meticulous processing steps, including filtering, illumination processing, and region segmentation, to accurately extract the second target region containing the target assembly opening. Feature matching is then performed using a feature vector dataset generated by a pre-trained deep learning model to filter out the two-dimensional contour feature point pixel information of the target assembly opening. Through a series of operations such as preprocessing, projection processing, and normalization of the three-dimensional visual image, an accurate data mapping relationship between the planar and three-dimensional visual images is established. This maps the two-dimensional contour feature point pixel information into three-dimensional spatial data. Finally, based on the obtained three-dimensional spatial data of the target assembly opening, the robotic arm is controlled to precisely execute the assembly operation of the tunnel segments to be assembled. This achieves precise control of the tunnel segment positioning and installation process, effectively improving the accuracy and efficiency of tunnel segment assembly, reducing human error, and ensuring the quality and safety of tunnel construction.

[0177] Figure 2 This is a schematic diagram of a tunnel segment positioning and installation system 100 based on image recognition, provided as an embodiment of this application. Figure 2 As shown, the processor 120 can be used in the image recognition-based tunnel segment positioning and installation system 100, and is used to perform the functions in this invention.

[0178] The image recognition-based tunnel segment positioning and installation system 100 can be a general-purpose server or a special-purpose server; both can be used to implement the image recognition-based tunnel segment positioning and installation method of this invention. Although only one server is shown in this invention, for convenience, the functions described in this invention can be implemented in a distributed manner on multiple similar platforms to balance the load.

[0179] For example, the image recognition-based tunnel segment positioning and installation system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and various forms of storage media 140, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the image recognition-based tunnel segment positioning and installation system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present invention can be implemented according to these program instructions. The image recognition-based tunnel segment positioning and installation system 100 also includes an input / output (I / O) interface 150 between a computer and other input / output devices.

[0180] For ease of explanation, only one processor is described in the image recognition-based tunnel segment positioning and installation system 100. However, it should be noted that the image recognition-based tunnel segment positioning and installation system 100 of the present invention may also include multiple processors, and therefore the steps performed by one processor described in the present invention may also be performed jointly by multiple processors or individually. For example, if the processor of the image recognition-based tunnel segment positioning and installation system 100 performs steps A and B, it should be understood that steps A and B may also be performed jointly by two different processors or individually by one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0181] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0182] Finally, it should be noted that the above-disclosed embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for positioning and installing tunnel segments based on image recognition, characterized in that, include: The location of the first target area, including the target assembly opening, is determined by acquiring three-dimensional scan images of the tunnel segment assembly surface using a three-dimensional scanning device deployed at the end of a robotic arm. Based on the location of the first target area, move the end of the robotic arm to the visual image acquisition position corresponding to the first target area; The planar visual image and the three-dimensional visual image of the first target area are acquired by a vision device deployed at the end of the robotic arm. The accuracy of the planar visual image and the three-dimensional visual image is higher than that of the three-dimensional scanned image. The installation position of the vision device at the end of the robotic arm is lower than that of the three-dimensional scanned image at the end of the robotic arm. The field of view of the vision device is smaller than that of the three-dimensional scanned image. Based on the planar visual image and the three-dimensional visual image, the three-dimensional spatial data of the target assembly port is obtained; Based on the three-dimensional spatial data of the target assembly port, the robotic arm is controlled to perform the assembly operation of the tunnel segments to be assembled.

2. The tunnel segment positioning and installation method based on image recognition according to claim 1, characterized in that, The process of acquiring three-dimensional scan images of the tunnel segment assembly surface using a three-dimensional scanning device deployed at the end of a robotic arm, and determining the location of a first target area including the target assembly opening, includes: The three-dimensional scanned image is subjected to image feature extraction processing to obtain composite spatial features included in the three-dimensional scanned image. The composite spatial features include at least one of edge contour features, geometric shape features, and surface texture features. The composite spatial features and the spatial structural features of the target assembly port are subjected to feature matching processing to determine whether the three-dimensional scanned image includes the first target region. If the first target region is not included in the three-dimensional scan image, the robotic arm is controlled to move and scan repeatedly based on a preset time interval and a preset scan path until the first target region is included in the three-dimensional scan image. The position of the first target region is determined based on the current position of the robotic arm and the position of the first target region in the three-dimensional scan image.

3. The tunnel segment positioning and installation method based on image recognition according to claim 1, characterized in that, The step of obtaining the three-dimensional spatial data of the target assembly port based on the planar visual image and the three-dimensional visual image includes: The planar visual image is filtered to obtain a filtered planar visual image; Based on the target detection threshold determined by using the edge of the tunnel segment joint as the threshold dividing line, the filtered planar visual image is subjected to region segmentation processing to remove invalid data regions in the filtered planar visual image and retain the second target region in the filtered planar visual image that includes the target assembly port. Obtain the target assembly port feature vector dataset generated by the pre-trained deep learning model; The feature data information matching process is performed based on the target assembly port feature vector dataset and the second target region, and the two-dimensional contour feature point pixel information of the target assembly port is filtered out from the second target region based on the matching confidence of the feature data information matching process. Based on the data mapping relationship between the planar visual image and the three-dimensional visual image, the pixel information of the two-dimensional contour feature points is transformed by three-dimensional spatial coordinate mapping to obtain the three-dimensional spatial data of the target assembly port.

4. The tunnel segment positioning and installation method based on image recognition according to claim 3, characterized in that, The process of performing a three-dimensional spatial coordinate mapping transformation on the pixel information of the two-dimensional contour feature points based on the data mapping relationship between the planar visual image and the three-dimensional visual image to obtain the three-dimensional spatial data of the target assembly port includes: A preprocessing operation is performed on a 3D visual image to obtain a preprocessed 3D visual image. The preprocessing operation includes at least one of the following: noise reduction processing, image enhancement processing, and downsampling processing. Based on the difference in field of view deflection between the preprocessed 3D visual image and the planar visual image, the preprocessed 3D visual image is projected to obtain a projected 3D visual image. The difference in field of view deflection between the projected 3D visual image and the planar visual image is less than or equal to a preset difference threshold. Based on the pixel coordinate scale of the planar visual image, the three-dimensional visual image after projection processing is normalized to obtain a normalized three-dimensional visual image. Based on the data mapping relationship between the planar visual image and the normalized three-dimensional visual image, the pixel information of the two-dimensional contour feature points is transformed by three-dimensional spatial coordinate mapping to obtain the three-dimensional spatial data of the target assembly port.

5. The tunnel segment positioning and installation method based on image recognition according to claim 4, characterized in that, The step of projecting the preprocessed 3D visual image based on the difference in field of view deflection between the preprocessed 3D visual image and the planar visual image to obtain a projected 3D visual image includes: Extract the target feature point set from the preprocessed 3D visual image. The target feature point set includes edge feature points of tunnel segments and splicing seam feature points. The edge feature points include continuous points on the outline of the tunnel segments, and the splicing seam feature points include the endpoints and midpoints of the gaps between adjacent tunnel segments. Extract the set of target feature points from the planar visual image. The set of target feature points corresponds to the same tunnel segment edges and splicing seams as the set of target feature points in the preprocessed three-dimensional visual image. By comparing the relative positional relationship between the target feature point set in the preprocessed 3D visual image and the target feature point set in the planar visual image, descriptive information of the field of view deflection difference is obtained. The descriptive information includes the offset of the target feature point in the horizontal direction, the offset of the target feature point in the vertical direction, and the rotation angle of the target feature point. Based on the description information of the difference in field of view, a projection adjustment rule is generated. The projection adjustment rule includes a first parameter for correcting the horizontal offset, a second parameter for correcting the vertical offset, and a third parameter for correcting the rotation angle. The projection adjustment rules are applied to the preprocessed 3D visual image to perform a projection transformation operation to obtain the projected 3D visual image. The projection transformation operation includes adjusting the horizontal coordinates of each point in the 3D visual image according to the first parameter, adjusting the vertical coordinates of each point in the 3D visual image according to the second parameter, and rotating each point in the 3D visual image according to the third parameter.

6. The tunnel segment positioning and installation method based on image recognition according to claim 4, characterized in that, The step of normalizing the projected 3D visual image based on the pixel coordinate scale of the planar visual image to obtain a normalized 3D visual image includes: Extract the pixel coordinate scale information of the planar visual image. The pixel coordinate scale information includes the horizontal spacing and vertical spacing of the pixels. The horizontal spacing is the distance between two adjacent pixels in the horizontal direction, and the vertical spacing is the distance between two adjacent pixels in the vertical direction. Extract the coordinate scale information of the three-dimensional visual image after projection processing. The coordinate scale information includes the horizontal unit length and the vertical unit length of the three-dimensional coordinates. The horizontal unit length is the actual distance between two adjacent coordinate points in the horizontal direction in the three-dimensional visual image, and the vertical unit length is the actual distance between two adjacent coordinate points in the vertical direction in the three-dimensional visual image. By comparing the pixel coordinate scale information of the planar visual image with the coordinate scale information of the three-dimensional visual image after projection processing, scale difference description information is obtained, which includes the proportional relationship of the horizontal scale and the proportional relationship of the vertical scale. Based on the scale difference description information, a scale adjustment rule is generated, which includes a horizontal scale parameter for adjusting the horizontal scale and a vertical scale parameter for adjusting the vertical scale. The coordinate scale of the projected 3D visual image is adjusted by applying the scale adjustment rules. The adjustment operation includes scaling the horizontal coordinate values ​​of each point in the 3D visual image according to the horizontal scale ratio parameter, and scaling the vertical coordinate values ​​of each point in the 3D visual image according to the vertical scale ratio parameter. Verify whether the coordinate scale of the adjusted 3D visual image is consistent with the pixel coordinate scale of the planar visual image; If they match, the adjusted 3D visual image is determined as the normalized 3D visual image. If the scales are inconsistent, adjust the horizontal scale parameter and / or the vertical scale parameter in the scale adjustment rule, and repeat the operation of applying the scale adjustment rule and verifying the scale until the scales are consistent.

7. The tunnel segment positioning and installation method based on image recognition according to claim 3, characterized in that, After filtering the planar visual image to obtain the filtered planar visual image, the method further includes: Extract the illumination distribution features of the filtered planar visual image, wherein the illumination distribution features include the brightness value distribution of each pixel in the filtered planar visual image; The filtered planar visual image is divided into regions based on the illumination distribution characteristics to obtain highlight regions, low-light regions, and normal illumination regions. The highlight regions are regions with brightness values ​​higher than a first preset range, the low-light regions are regions with brightness values ​​lower than a second preset range, and the normal illumination regions are regions with brightness values ​​between the first preset range and the second preset range. Adaptive gamma compression processing is performed on the highlight area. The adaptive gamma compression processing includes determining gamma compression parameters based on the average brightness value of the highlight area, and using the gamma compression parameters to reduce the brightness of overexposed pixels in the highlight area. Adaptive histogram equalization is performed on the low-light region. The adaptive histogram equalization process includes determining the range of histogram equalization based on the pixel brightness distribution of the low-light region, and using the range to improve the contrast of dark details in the low-light region. The processed highlight area, low light area and normal lighting area are merged at the pixel level to obtain a light-enhanced planar visual image. The region segmentation process is performed based on the illuminated planar visual image.

8. The tunnel segment positioning and installation method based on image recognition according to claim 3, characterized in that, The step of performing region segmentation processing on the filtered planar visual image based on the target detection threshold determined by using the edge of the tunnel segment joint as the threshold dividing line, removing invalid data regions from the filtered planar visual image, and retaining the second target region in the filtered planar visual image that includes the target assembly opening, includes: Extract the gradient features of the tunnel segment joint edges in the filtered planar visual image, wherein the gradient features include the grayscale change rate of pixels at the tunnel segment joint edges; Calculate the mean and variance of the gradient features of the tunnel segment joint edge, where the mean is the average of the gradient values ​​of all pixels at the tunnel segment joint edge, and the variance is the average of the sum of squared deviations of the gradient values ​​of all pixels at the tunnel segment joint edge from the mean. The target detection threshold is dynamically adjusted based on the mean and variance of the gradient features. If the variance of the gradient features is greater than or equal to a preset variance value, the target detection threshold is reduced to retain more edge details. If the variance of the gradient features is less than the preset variance value, the target detection threshold is increased to remove noise. Multiple sampling points are selected at the edge of the tunnel segment joint, and the multiple sampling points are evenly distributed at different positions on the edge of the tunnel segment joint. Check whether the edges after threshold segmentation at each sampling point are continuous, and whether the edges between adjacent sampling points are continuous, in order to verify the continuity of the edges after the adjusted target detection threshold segmentation; If the edge continuity is insufficient, the target detection threshold is fine-tuned again, and the fine-tuning includes adjusting the size of the target detection threshold according to the degree of continuity insufficiency; The filtered planar visual image is subjected to the region segmentation process based on the final adjusted target detection threshold.

9. The tunnel segment positioning and installation method based on image recognition according to claim 4, characterized in that, The method of performing a three-dimensional spatial coordinate mapping transformation on the pixel information of the two-dimensional contour feature points based on the data mapping relationship between the planar visual image and the normalized three-dimensional visual image to obtain the three-dimensional spatial data of the target assembly port includes: Acquire attitude sensor data at the end of the robotic arm, the attitude sensor data including the position coordinates, pitch angle, yaw angle, and roll angle of the end of the robotic arm; The acquisition viewpoint offset of the vision device is calculated based on the attitude sensor data, and the acquisition viewpoint offset includes the tilt angle of the vision device relative to the preset acquisition viewpoint. The acquisition viewpoint offset is incorporated into the mapping matrix between the planar visual image and the normalized 3D visual image to obtain a corrected mapping matrix. The incorporation includes converting the acquisition viewpoint offset into a rotation matrix and multiplying the rotation matrix by the original mapping matrix. Based on the modified mapping matrix, the pixel information of the two-dimensional contour feature points is transformed into three-dimensional spatial coordinates to obtain the three-dimensional spatial data of the target assembly port. The transformation includes substituting the pixel coordinates of the pixel information of the two-dimensional contour feature points into the modified mapping matrix to obtain the corresponding three-dimensional spatial coordinates.

10. A tunnel segment positioning and installation system based on image recognition, characterized in that, The method includes a processor and a computer-readable storage medium storing machine-executable instructions, which, when executed by a computer, implement the tunnel segment positioning and installation method based on image recognition as described in any one of claims 1-9.

Images