Tunnel video stream three-dimensional modeling system based on binocular stereo matching and SLAM

Abstract

The invention relates to the technical field of computer vision, and particularly provides a tunnel video stream three-dimensional modeling system based on binocular stereo matching and SLAM (Simultaneous Localization and Mapping), which comprises a multi-modal image acquisition unit, a binocular infrared camera and an RGB (Red, Green and Blue) camera are configured to synchronously acquire an infrared image pair, an RGB image pair and inertial measurement unit data of a tunnel environment, the multi-source data is uploaded to the cloud processing platform in real time through the wireless transmission module; based on a dynamic switching mechanism of environmental perception, performing adaptive fusion pose estimation of a feature point method and a direct method on input multi-modal image data; the stereo matching and dense reconstruction unit is used for calculating a disparity map by adopting an improved self-adaptive window stereo matching algorithm, generating a three-dimensional point cloud in combination with the pose information output by the pose estimation unit, and constructing a global consistency point cloud model of the tunnel scene through a time sequence point cloud registration and splicing algorithm; and a multi-algorithm target detection and semantic fusion unit. The method can meet the requirements of precision and robustness of tunnel reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM. Background Technology

[0002] The rational development and utilization of urban underground space can provide new opportunities for reducing environmental pollution, increasing green areas, and reducing and sequestering carbon emissions. Tunnels are an indispensable and important component of underground space, playing a vital role as a medium for transporting people and materials, and are an important way for humans to expand their activity space and improve their quality of life.

[0003] With the continuous and rapid development of underground space in my country, the frequency of various accidents and disasters during the construction and operation of underground spaces has increased dramatically. Among these, tunnel environments are particularly vulnerable to roof collapses, fires, and other accidents, which can easily lead to casualties. Automated 3D tunnel reconstruction can improve the efficiency of daily tunnel maintenance, promptly identify defects, and prevent potential disasters. In the event of a disaster, to improve emergency response speed and rescue efficiency, it is equally necessary to provide rescue robots and personnel with 3D tunnel reconstruction capabilities, enabling them to quickly understand the spatial status of the tunnel, the distribution of personnel and facilities, and thus conduct rapid and scientific rescue operations.

[0004] Currently, 3D reconstruction solutions for tunnel scenes can be divided into three types: manual modeling based on 3D software (such as 3dsMAX, Maya, Cinema4D); vision-based 3D reconstruction using multi-view geometry or deep learning methods; and point cloud 3D reconstruction based on LiDAR.

[0005] The above reconstruction plans all have certain limitations: Limitations of manual modeling using 3D software (such as 3ds Max, Maya, Cinema 4D): Low mapping efficiency: It requires a large amount of manpower and takes a long time to model; Prerequisite information: Information about the scene to be reconstructed needs to be known in advance; High skill requirements: Requires a high level of skill and experience to ensure the quality and detail of the model; Lack of location information: Unable to obtain the location information of objects in the scene; Limitations of vision-based 3D reconstruction using methods such as multi-view geometry or deep learning: Time and perspective limitations: The quality of tunnel scene images is poor and there are few images. Multi-view geometry methods need to acquire images from different perspectives. If there are not enough perspectives or the differences between perspectives are not significant, it may be impossible to accurately reconstruct the 3D structure, and the time consumption is extremely high.

[0006] Training data dependency: Currently, training sets for tunnel scenarios are extremely limited, and the effectiveness of deep learning methods largely depends on the quality and diversity of training data. If the training data is insufficient or biased, the model may fail to generalize to new data.

[0007] To address the shortcomings of the existing technology, the present invention provides a 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM. Summary of the Invention

[0008] To achieve the above objectives, the present invention adopts the following technical solution: In one aspect, the present invention provides a 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM, comprising: The multimodal image acquisition unit is equipped with a binocular infrared camera and an RGB camera to simultaneously acquire infrared image pairs, RGB image pairs, and inertial measurement unit data of the tunnel environment, and uploads the multi-source data to the cloud processing platform in real time through a wireless transmission module. The visual-inertial pose estimation unit, based on a dynamic switching mechanism of environmental perception, performs adaptive fusion pose estimation of the input multimodal image data using the feature point method and the direct method: when a weak texture region of tunnel is detected, it switches to the improved direct method to improve robustness; when a texture-rich region is detected, it switches to the optimized feature point method to improve accuracy. It also achieves scale calibration and motion distortion correction by tightly coupling IMU data, and outputs continuous camera trajectory and inter-frame pose relationship. The stereo matching and dense reconstruction unit uses an improved adaptive window stereo matching algorithm to calculate the disparity map, combines the pose information output by the pose estimation unit to generate a 3D point cloud, and constructs a globally consistent point cloud model of the tunnel scene through a temporal point cloud registration and stitching algorithm. The multi-algorithm object detection and semantic fusion unit integrates an extensible algorithm library. It calls the object detection model to detect RGB images according to user configuration, semantically associates the two-dimensional detection box with the three-dimensional point cloud through spatial coordinate mapping, and realizes the dynamic overlay of the detection results and the point cloud model based on the confidence weighted fusion strategy to generate a three-dimensional scene with semantic annotation.

[0009] In one optional implementation, the improved adaptive window stereo matching algorithm includes: The traditional Census transform is improved by constructing feature vectors through the extreme and mean gradient values ​​of pixels within the window in four directions: horizontal, vertical, 45°, and 135°. Three-bit binary encoding is used to realize multi-directional description of the pixel neighborhood structure. The size and shape of the matching window are dynamically adjusted by combining the pixel gradient magnitude and texture entropy value. Elliptical expansion windows are used in weak texture areas, and square contraction windows are used in strong texture areas.

[0010] In one optional implementation, the dynamic switching mechanism of the visual-inertial pose estimation unit specifically includes: The environmental texture evaluation module quantifies the richness of environmental texture by calculating the number of FAST corner points, SIFT feature point density, and grayscale variance of the current frame image. The decision switcher triggers the direct method for pose estimation when the texture evaluation value is below a set threshold for three consecutive frames; when the texture evaluation value rises back above the threshold, it automatically switches back to the ORB-SLAM feature point method and achieves a smooth transition between the two methods through keyframe relocalization.

[0011] In an optional implementation, the multi-algorithm object detection and semantic fusion unit further includes: The algorithm scheduler allows users to upload custom detection models via API and dynamically allocate GPU computing resources based on hardware resource utilization. The spatiotemporal consistency verification module performs trajectory prediction and IOU matching on the target detection results of consecutive frames, removes false detection boxes, and interpolates and completes the missed targets, thereby improving the spatiotemporal continuity of semantic annotation.

[0012] In one optional implementation, the stereo matching and dense reconstruction unit further includes a frame selection and dynamic reconstruction scheduling mechanism: based on the motion speed and rotation angle output by the pose estimation unit, when the device motion speed > threshold V1 or the rotation angle > threshold θ, the point cloud reconstruction frame rate is reduced; when the motion is stable, the reconstruction frame rate is increased to balance real-time performance and modeling accuracy.

[0013] In one optional implementation, the specific steps of the multi-directional Census encoding are as follows: For each pixel within the matching window, calculate its maximum gradient value Gmax and minimum gradient value Gmin in the four directions; If the current pixel gradient value > Gmax × 0.8, then the code is 11; if Gmin × 0.2 < gradient value ≤ Gmax × 0.8, then the code is 10; if the gradient value ≤ Gmin × 0.2, then the code is 01; if the gradient value is zero, then the code is 00. The codes from the four directions are concatenated into a 12-bit binary code, which serves as the Census feature descriptor for that pixel.

[0014] In one optional implementation, the multimodal image acquisition unit further includes an infrared-RGB image registration module, which obtains the extrinsic parameter matrices of the infrared camera and the RGB camera through Zhang's calibration method and hand-eye coordination calibration, and superimposes the temperature information of the infrared image onto the corresponding area of ​​the RGB image through projection transformation to assist the cross-modal feature learning of the target detection model.

[0015] In one optional implementation, the semantic fusion strategy specifically includes: For pixels within the 2D detection frame, their 3D coordinates are calculated using the in-camera parallax map. The DBSCAN clustering algorithm is used to cluster the 3D coordinate point cloud and remove outliers; Different color labels are assigned to the categories of the detected targets, and they are displayed in the point cloud model as semi-transparent bounding boxes. The size of the bounding boxes is dynamically adjusted based on prior knowledge of the actual size of the targets.

[0016] In one optional implementation, the system also includes an edge-cloud collaborative computing module, which deploys lightweight feature extraction and coarse localization tasks on edge devices and deploys computationally intensive tasks such as dense reconstruction, global optimization, and semantic fusion on the cloud, thereby achieving low-latency data transmission and task scheduling through 5G slicing technology.

[0017] In one optional implementation, the temporal point cloud registration and stitching algorithm adopts an improved ICP algorithm: combining the structural prior of the tunnel scene, a directional constraint term is added to the point-to-point distance loss function of the traditional ICP. When the angle between the registration error vector and the tunnel axis is greater than the threshold α, a penalty weight is added to avoid model distortion caused by accumulated errors.

[0018] Effects of the invention: By using a robust SLAM scheme and a high-precision stereo matching algorithm, the challenges of weak texture and missing features in tunnel scenes are overcome. A reconstruction frame selection strategy is designed to effectively depict scene details in real time while ensuring the robustness of mapping, thus meeting the accuracy and robustness requirements of tunnel reconstruction.

[0019] It can provide location information to assist in precise rescue. Because it uses a combination of SLAM and stereo matching, the location of people can be located through SLAM during the mapping process, and the location coordinate information of specific people or objects can be obtained in the complete 3D map.

[0020] It can meet the needs of different users and different scenarios. It provides configurable target detection templates, allowing users to customize the detection algorithms they need and set the fusion strategy with 3D maps.

[0021] It has a wide range of applications, is fully automated, requires no prior information, can build maps in unknown environments, and has greater versatility. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1This is a framework diagram of a tunnel video stream 3D modeling system based on binocular stereo matching and SLAM provided in Embodiment 1 of the present invention; Figure 2 This is a diagram illustrating the overall framework of the stereo matching algorithm according to an embodiment of the present invention. Figure 3 This is an improved overall flowchart of the Census transform according to an embodiment of the present invention; Figure 4 This is a block diagram of the electronic device provided in Embodiment 3 of the present invention; Figure 5 This is a block diagram of a computer-readable storage medium provided in Embodiment 3 of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0024] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0025] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise explicitly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0026] In this embodiment of the invention, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0027] Example 1: like Figure 1 , Figure 2 , Figure 3 As shown, this embodiment of the invention provides a 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM, comprising: The multimodal image acquisition unit is equipped with a binocular infrared camera and an RGB camera to simultaneously acquire infrared image pairs, RGB image pairs, and inertial measurement unit data of the tunnel environment, and uploads the multi-source data to the cloud processing platform in real time through a wireless transmission module. The visual-inertial pose estimation unit, based on a dynamic switching mechanism of environmental perception, performs adaptive fusion pose estimation of the input multimodal image data using the feature point method and the direct method: when a weak texture region of tunnel is detected, it switches to the improved direct method to improve robustness; when a texture-rich region is detected, it switches to the optimized feature point method to improve accuracy. It also achieves scale calibration and motion distortion correction by tightly coupling IMU data, and outputs continuous camera trajectory and inter-frame pose relationship. The stereo matching and dense reconstruction unit uses an improved adaptive window stereo matching algorithm to calculate the disparity map, combines the pose information output by the pose estimation unit to generate a 3D point cloud, and constructs a globally consistent point cloud model of the tunnel scene through a temporal point cloud registration and stitching algorithm. The multi-algorithm object detection and semantic fusion unit integrates an extensible algorithm library. It calls the object detection model to detect RGB images according to user configuration, semantically associates the two-dimensional detection box with the three-dimensional point cloud through spatial coordinate mapping, and realizes the dynamic overlay of the detection results and the point cloud model based on the confidence weighted fusion strategy to generate a three-dimensional scene with semantic annotation.

[0028] In the above embodiments, the overall system architecture can be summarized into four main parts: information acquisition, pose estimation, target detection and point cloud fusion, and 3D reconstruction. First, the information acquisition part acquires binocular infrared and RGB images using a binocular infrared camera and an RGB camera, and transmits them wirelessly to the Shenyan video platform in real time. Then, relying on cloud computing power, the information is processed in real time. The pose estimation part uses the binocular infrared and RGB images sent by the acquisition device as input to the pose estimation module to calculate the pose information of the current frame. Simultaneously, the reconstruction module determines whether the current frame needs point cloud computing based on a reconstruction frame selection strategy. If point cloud computing is required, the image information and pose information of that frame are sent to the reconstruction queue, and the image information is synchronously transmitted to the target detection module for detection according to the configured algorithm library. The target detection module then fuses the reconstructed information with the target detection information using a set fusion strategy, such as for personnel and flame detection. In the reconstruction module, the reconstructed information and target detection information are fused to calculate the 3D point cloud of that frame, which is then stitched together with the previous reconstructed frame to form a complete scene 3D point cloud.

[0029] Example 2: like Figure 1 As shown, based on Embodiment 1, the improved adaptive window stereo matching algorithm of this invention includes: The traditional Census transform is improved by constructing feature vectors through the extreme and mean gradient values ​​of pixels within the window in four directions: horizontal, vertical, 45°, and 135°. Three-bit binary encoding is used to realize multi-directional description of the pixel neighborhood structure. The size and shape of the matching window are dynamically adjusted by combining the pixel gradient magnitude and texture entropy value. Elliptical expansion windows are used in weak texture areas, and square contraction windows are used in strong texture areas.

[0030] Specifically, the dynamic switching mechanism of the visual-inertial pose estimation unit includes: The environmental texture evaluation module quantifies the richness of environmental texture by calculating the number of FAST corner points, SIFT feature point density, and grayscale variance of the current frame image. The decision switcher triggers the direct method for pose estimation when the texture evaluation value is below a set threshold for three consecutive frames; when the texture evaluation value rises back above the threshold, it automatically switches back to the ORB-SLAM feature point method and achieves a smooth transition between the two methods through keyframe relocalization.

[0031] Specifically, the multi-algorithm object detection and semantic fusion unit further includes: The algorithm scheduler allows users to upload custom detection models via API and dynamically allocate GPU computing resources based on hardware resource utilization. The spatiotemporal consistency verification module performs trajectory prediction and IOU matching on the target detection results of consecutive frames, removes false detection boxes, and interpolates and completes the missed targets, thereby improving the spatiotemporal continuity of semantic annotation.

[0032] Specifically, the stereo matching and dense reconstruction unit also includes a frame selection and dynamic reconstruction scheduling mechanism: based on the motion speed and rotation angle output by the pose estimation unit, when the device motion speed > threshold V1 or the rotation angle > threshold θ, the point cloud reconstruction frame rate is reduced; when the motion is stable, the reconstruction frame rate is increased to balance real-time performance and modeling accuracy.

[0033] Specifically, the steps of the multi-directional Census encoding are as follows: For each pixel within the matching window, calculate its maximum gradient value Gmax and minimum gradient value Gmin in the four directions; If the current pixel gradient value > Gmax × 0.8, then the code is 11; if Gmin × 0.2 < gradient value ≤ Gmax × 0.8, then the code is 10; if the gradient value ≤ Gmin × 0.2, then the code is 01; if the gradient value is zero, then the code is 00. The codes from the four directions are concatenated into a 12-bit binary code, which serves as the Census feature descriptor for that pixel.

[0034] Specifically, the multimodal image acquisition unit also includes an infrared-RGB image registration module, which obtains the extrinsic parameter matrices of the infrared camera and the RGB camera through Zhang's calibration method and hand-eye coordination calibration, and superimposes the temperature information of the infrared image onto the corresponding area of ​​the RGB image through projection transformation to assist the cross-modal feature learning of the target detection model.

[0035] Specifically, the semantic fusion strategy is as follows: For pixels within the 2D detection frame, their 3D coordinates are calculated using the in-camera parallax map. The DBSCAN clustering algorithm is used to cluster the 3D coordinate point cloud and remove outliers; Different color labels are assigned to the categories of the detected targets, and they are displayed in the point cloud model as semi-transparent bounding boxes. The size of the bounding boxes is dynamically adjusted based on prior knowledge of the actual size of the targets.

[0036] Specifically, it also includes an edge-cloud collaborative computing module, which deploys lightweight feature extraction and coarse localization tasks on edge devices, and deploys computationally intensive tasks such as dense reconstruction, global optimization and semantic fusion on the cloud, and achieves low-latency data transmission and task scheduling through 5G slicing technology.

[0037] Specifically, the temporal point cloud registration and stitching algorithm adopts an improved ICP algorithm: combining the structural prior of the tunnel scene, a directional constraint term is added to the point-to-point distance loss function of the traditional ICP. When the angle between the registration error vector and the tunnel axis is greater than the threshold α, a penalty weight is added to avoid model distortion caused by accumulated errors.

[0038] In the above embodiments, to meet the requirements of 3D reconstruction of tunnel scenes, the pose estimation part should have high robustness and ensure stable pose output in harsh environments to meet the positioning requirements of subsequent 3D reconstruction. Binocular stereo matching is the core of the 3D reconstruction part, and it should have the ability to perform stereo matching in weakly textured areas of the tunnel to ensure the integrity of the reconstruction. Below, I will provide a detailed introduction to the two main parts, stereo matching and pose estimation, and also introduce the combination of target detection and 3D reconstruction.

[0039] Design of a binocular stereo matching system: Compared with ordinary scenes, stereo matching in tunnel scenes presents the following challenges: (1) uneven illumination in tunnels and excessive exposure time of visual sensors lead to increased salt-and-pepper noise in the images; (2) tunnel images have weak texture and repetitive texture. The essence of stereo matching is to find corresponding points in two images. Salt-and-pepper noise and weak texture will cause ambiguity in the matching of corresponding points, which will seriously reduce the matching accuracy. Census transform is not sensitive to the overall brightness and darkness of the image and has a certain ability to process weak textures, so it is widely used. However, it is sensitive to noise and easily ignores specific gray values.

[0040] To address the aforementioned challenges, a stereo binocular algorithm based on AD-Census was designed, with the overall architecture as follows: Figure 2 As shown, it can be mainly divided into the following two parts: 1) Improvement of Census algorithm; 2) Adaptive fusion of Census and SAD.

[0041] First, the noise resistance of the algorithm is improved by refining the Census encoding method. Then, an adaptive Census window is designed to improve the algorithm's ability to handle textured regions, resulting in an improved overall workflow.

[0042] First, the Census encoding method is improved to reduce its dependence on the center pixel, thereby enhancing the algorithm's noise resistance. Specific improvements include... Figure 4 As shown. In the cost calculation window, the average value of all pixels is calculated. Calculate with the center pixel The average grayscale value of all pixels with the origin as the axis and the horizontal, vertical, left diagonal, and right diagonal directions as the axes. Let the maximum and minimum values ​​be denoted as . and The center pixel is compared with all its neighboring pixels within the cost calculation window. and Compare them; if greater than The code is 01 if it is less than 1. The code is 10 if it is 10, and 00 otherwise.

[0043] The improved Census encoding method described above also uses binary encoding, but it differs in that it employs three different encoding formats using two binary digits to encode neighboring pixels, thus improving overall information utilization. This encoding method can adaptively classify neighboring pixels into three categories for separate encoding, and the threshold affecting the encoding is less affected by the center pixel, reducing dependence on the center pixel and improving the algorithm's noise resistance and robustness.

[0044] The tunnel scene has many areas with weak textures, but there are also areas with clear texture structures, such as... Figure 5 As shown, the proportion of weak texture regions varies in different images. To improve the processing effect of weak textures, different cost calculation windows should be used.

[0045] First, calculate the gradient of each pixel in the image. gradient value Less than If a pixel is not found to be a weak texture pixel, then the proportion of weak texture pixels among all pixels is calculated. Then the ratio of weak texture pixels The weak texture point ratio threshold is set for different window sizes. , Compare and determine which window size to use: (1) The Census transform is a non-parametric transform that only utilizes the relative grayscale relationships between neighboring pixels within the cost window, without considering the specific grayscale values. This can lead to suboptimal matching results in texture-rich regions, thus easily degrading tunnel reconstruction quality. Fusion with the SAD method can effectively solve this problem.

[0046] First, an adaptive window is constructed for SAD, but unlike the improved Census adaptive window strategy, a pixel-level adaptive window strategy is adopted to improve SAD. The smaller the gradient of a pixel, the smoother the grayscale information change around it, and the similar the colors; conversely, the larger the gradient, the more obvious the grayscale change around it, and the greater the color difference. Based on this fact, different cost calculation windows are assigned according to the gradient differences between pixels, as shown in the formula: (2) in, , The set gradient threshold; Then, calculate the SAD cost: (3) in, For pixels At parallax The cost window is The cost of SAD at that time; , The cost window is respectively The number of columns and rows in the window.

[0047] Finally, the Census cost and SAD cost calculation results are fused. Since the improved Census transform cost and the improved Census cost have different scales, they need to be normalized and scale-aligned before fusion, followed by weighted fusion. Therefore, this invention uses a natural exponential function to normalize the two different scales of costs to a unified scale, and adaptively adjusts the weights of the two matching costs based on the proportion of weak texture pixels in the image. The calculation formula is as follows: (4) in, For control parameters, For the sake of value, through control Adjustable The degree of sensitivity. When , When both are positive, Normalization is possible The range is [0, 1]. The final cost fusion formula can be expressed as: (5) in, For pixels At parallax The cost of integration at that time; To improve the Census transform in The value of a point To improve SAD in The value of a point; To improve the Census cost control parameters, To improve the cost control parameters of SAD; among which The specific calculation method is as follows: (6) in, The proportion of weak texture points calculated by equation (12) for Cost control parameters.

[0048] The unique characteristics of tunnel environments cause feature point-based odometry to suffer from feature loss, leading to pose tracking failure. If odometry positioning is interrupted, 3D point cloud stitching based on camera position cannot be achieved, ultimately resulting in 3D reconstruction failure. Therefore, a highly robust odometry with good positioning accuracy is necessary. This invention addresses these challenges by employing a visual-inertial odometry method that combines feature point and direct methods, building upon ORB_SLAM3.

[0049] In tunnel scenarios, the motion state and environmental characteristics of mobile robots lead to a high probability of ORB_SLAM3 relocalization failure, and the robustness of the algorithm's tracking cannot be guaranteed. In this invention, when the number of matching pairs between two frames is less than a threshold... At this time, instead of expanding the search range, a direct method is used for pose estimation to ensure the continuity of tracking. In the invention... Set to 30. In vision-based odometry, although feature-point-based pose estimation is significantly affected by the environment, it still boasts the highest accuracy. Therefore, feature-point-based initialization is still used during startup, only reducing the number of matching pairs during operation. In some cases, a direct method is used for tracking. However, to achieve consistency in pose estimation, when the number of feature matching pairs is less than [a certain value], [the following method is recommended]. The time-pre-start direct method, if the number of matching pairs during this period is less than The pose is then estimated using the direct method.

[0050] Conventional 3D reconstructed maps may not meet all user needs. For example, in tunnel emergency rescue scenarios, users are more concerned about trapped personnel and dangerous factors such as flames, while in routine maintenance, users are more concerned about personnel and vehicle behavior within the tunnel. Therefore, this invention uses a configurable algorithm detection library for users to configure detection algorithms and implements algorithm calls via API calls, reducing coupling.

[0051] After target detection is performed on the reconstructed frame, it is fused with the point cloud of that frame. Different fusion strategies can be set for different detection results. For example, in personnel detection, personnel entering the detection area can be automatically identified, and the total number of personnel in the image can be counted. If the number exceeds the set total number of personnel, an early warning is issued. In this use case, the personnel threshold is set to 1, and the early warning is used as the processing signal. When an early warning signal is generated, the location of the detection box is obtained, and all point clouds within the detection box are marked as trapped personnel. For flame detection, it can be set to mark the point cloud of the boundary of the area instead of marking the point cloud of the entire area.

[0052] Since camera output frequencies are generally high, if the camera information output frequency is 30Hz, performing stereo matching on each image output by the camera to generate a 3D point cloud would not only waste computing resources but also cause point cloud redundancy. Therefore, it is necessary to design a reasonable reconstruction frame selection strategy.

[0053] A reconstruction frame selection strategy was designed by combining the movement distance and rotation angle.

[0054] Tunnels are noisy environments, and the effectiveness of noise resistance significantly impacts the final matching result. By adding different proportions of salt-and-pepper noise to the tunnel scene, it can be seen that the binocular stereo matching algorithm of this invention has high noise resistance and can meet the noise requirements for stereo matching in tunnel emergency rescue scenarios.

[0055] To further verify the noise robustness of the proposed algorithm, salt-and-pepper noise of 2%, 4%, 6%, and 8% was added to the Pipes standard data of the Middlebury platform, respectively, and the overall noise robustness of the proposed algorithm and current mainstream algorithms was analyzed. Specific analysis data are shown in Table 1.

[0056] Table 1. Mismatch rates of the three algorithms under different noise densities.

[0057] As shown in Table 1, the mismatch rates of all three algorithms increase with the increasing noise density of salt-and-pepper noise. However, the mismatch rate of the algorithm presented in this invention is lower than that of the other two algorithms at all noise densities. When the noise density is 0%-6%, the rate of increase in mismatches for the algorithm presented in this invention is lower than that for the other two algorithms. When the noise density is 6%-8%, all three algorithms have relatively high mismatch rates, but the mismatch rate of the algorithm presented in this invention is still lower than that of the other two algorithms, demonstrating higher robustness to noise.

[0058] The algorithm of this invention has good processing effects on small targets, weak texture regions, and image edge regions in images. It can cope with the challenge of having many weak texture regions in tunnel scenes.

[0059] In areas with few feature points and many weak texture regions, the odometry of this invention can still provide robust pose output, providing pose information for subsequent reconstruction, ensuring simultaneous localization and mapping, and meeting the pose output requirements for tunnel emergency rescue.

[0060] With the design described above, we can perform 3D reconstruction of tunnel scenes in harsh environments. A practical application example is given below.

[0061] Table 2 Reconstruction Data Analysis

[0062] The experimental results above show that the proposed algorithm achieves a reconstruction accuracy error within 3%, demonstrating good precision and meeting the needs of practical tunnel scene mapping. In terms of reconstruction detail, it clearly depicts object details with high recognizability. Regarding mapping robustness, the algorithm successfully constructed a complete 130m-long 3D tunnel scene, exhibiting good robustness.

[0063] Example 3 Figure 4 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown.

[0064] The electronic device may include a central processing unit / microprocessor / main control chip, etc. 4; and a storage medium 5, coupled to the central processing unit / microprocessor / main control chip, etc. 4, and storing computer-executable instructions therein for performing the steps of various methods of embodiments of the present invention when executed by the processor.

[0065] The central processing unit / microprocessor / main control chip, etc., can include, but are not limited to, one or more processors or microprocessors.

[0066] Storage medium 5 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0067] In addition, the electronic device may also include (but is not limited to) a data bus 6, an input / output bus / external bus / device bus 7, a display 8, and input / output devices 9 (e.g., keyboard, mouse, speaker, etc.).

[0068] The central processing unit / microprocessor / main control chip, etc. 4 can communicate with external devices (8, 9, etc.) via I / O bus 7 through wired or wireless network (not shown).

[0069] The storage medium 5 may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip, etc., 4 is running.

[0070] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0071] Figure 5A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown.

[0072] like Figure 5 As shown, the non-transitory computer-readable storage medium 11 stores instructions, such as computer-readable instructions 10. When the computer-readable instructions 10 are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 11 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 10 stored on the computer-readable storage medium 11, the various methods described above can be performed.

[0073] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0074] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this invention through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM, characterized in that, include: The multimodal image acquisition unit is equipped with a binocular infrared camera and an RGB camera to simultaneously acquire infrared image pairs, RGB image pairs, and inertial measurement unit data of the tunnel environment, and uploads the multi-source data to the cloud processing platform in real time through a wireless transmission module. The visual-inertial pose estimation unit, based on a dynamic switching mechanism of environmental perception, performs adaptive fusion pose estimation of the input multimodal image data using the feature point method and the direct method: when a weak texture region of tunnel is detected, it switches to the improved direct method to improve robustness; when a texture-rich region is detected, it switches to the optimized feature point method to improve accuracy. It also achieves scale calibration and motion distortion correction by tightly coupling IMU data, and outputs continuous camera trajectory and inter-frame pose relationship. The stereo matching and dense reconstruction unit uses an improved adaptive window stereo matching algorithm to calculate the disparity map, combines the pose information output by the pose estimation unit to generate a 3D point cloud, and constructs a globally consistent point cloud model of the tunnel scene through a temporal point cloud registration and stitching algorithm. The multi-algorithm object detection and semantic fusion unit integrates an extensible algorithm library. It calls the object detection model to detect RGB images according to user configuration, semantically associates the two-dimensional detection box with the three-dimensional point cloud through spatial coordinate mapping, and realizes the dynamic overlay of the detection results and the point cloud model based on the confidence weighted fusion strategy to generate a three-dimensional scene with semantic annotation.

2. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The improved adaptive window stereo matching algorithm includes: The traditional Census transform is improved by constructing feature vectors through the extreme and mean gradient values ​​of pixels within the window in four directions: horizontal, vertical, 45°, and 135°. Three-bit binary encoding is used to realize multi-directional description of the pixel neighborhood structure. The size and shape of the matching window are dynamically adjusted by combining the pixel gradient magnitude and texture entropy value. Elliptical expansion windows are used in weak texture areas, and square contraction windows are used in strong texture areas.

3. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The dynamic switching mechanism of the visual-inertial pose estimation unit specifically includes: The environmental texture evaluation module quantifies the richness of environmental texture by calculating the number of FAST corner points, SIFT feature point density, and grayscale variance of the current frame image. The decision switcher triggers the direct method for pose estimation when the texture evaluation value is below a set threshold for three consecutive frames; when the texture evaluation value rises back above the threshold, it automatically switches back to the ORB-SLAM feature point method and achieves a smooth transition between the two methods through keyframe relocalization.

4. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The multi-algorithm target detection and semantic fusion unit also includes: The algorithm scheduler allows users to upload custom detection models via API and dynamically allocate GPU computing resources based on hardware resource utilization. The spatiotemporal consistency verification module performs trajectory prediction and IOU matching on the target detection results of consecutive frames, removes false detection boxes, and interpolates and completes the missed targets, thereby improving the spatiotemporal continuity of semantic annotation.

5. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The stereo matching and dense reconstruction unit also includes a frame selection and dynamic reconstruction scheduling mechanism: based on the motion speed and rotation angle output by the pose estimation unit, when the device motion speed > threshold V1 or the rotation angle > threshold θ, the point cloud reconstruction frame rate is reduced; when the motion is stable, the reconstruction frame rate is increased to balance real-time performance and modeling accuracy.

6. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 2, characterized in that, The specific steps of the multi-directional Census encoding are as follows: For each pixel within the matching window, calculate its maximum gradient value Gmax and minimum gradient value Gmin in the four directions; If the current pixel gradient value > Gmax × 0.8, then the code is 11; if Gmin × 0.2 < gradient value ≤ Gmax × 0.8, then the code is 10; if the gradient value ≤ Gmin × 0.2, then the code is 01; if the gradient value is zero, then the code is 00. The codes from the four directions are concatenated into a 12-bit binary code, which serves as the Census feature descriptor for that pixel.

7. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The multimodal image acquisition unit also includes an infrared-RGB image registration module, which obtains the extrinsic parameter matrices of the infrared camera and the RGB camera through Zhang's calibration method and hand-eye coordination calibration, and superimposes the temperature information of the infrared image onto the corresponding area of ​​the RGB image through projection transformation to assist the cross-modal feature learning of the target detection model.

8. The tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The semantic fusion strategy is specifically as follows: For pixels within the 2D detection frame, their 3D coordinates are calculated using the in-camera disparity map. The DBSCAN clustering algorithm is used to cluster the 3D coordinate point cloud and remove outliers; Different color labels are assigned to the categories of the detected targets, and they are displayed in the point cloud model as semi-transparent bounding boxes. The size of the bounding boxes is dynamically adjusted based on prior knowledge of the actual size of the targets.

9. A tunnel video stream 3D modeling system based on binocular stereo matching and SLAM as described in claim 1, characterized in that, It also includes an edge-cloud collaborative computing module, which deploys lightweight feature extraction and coarse localization tasks on edge devices, and deploys computationally intensive tasks such as dense reconstruction, global optimization and semantic fusion in the cloud, and achieves low-latency data transmission and task scheduling through 5G slicing technology.

10. A 3D modeling system for tunnel video streams based on binocular stereo matching and SLAM as described in claim 1, characterized in that, The temporal point cloud registration and stitching algorithm adopts an improved ICP algorithm: combining the structural prior of the tunnel scene, a directional constraint term is added to the point-to-point distance loss function of the traditional ICP. When the angle between the registration error vector and the tunnel axis is greater than the threshold α, a penalty weight is added to avoid model distortion caused by accumulated errors.

Images