Light-weight working face three-dimensional reconstruction method and device fusing geometric prior constraints

Abstract

The invention relates to the technical field of computer vision and coal mine intelligent mining, and discloses a lightweight working face three-dimensional reconstruction method and device fusing geometric prior constraints, and the method comprises the steps: firstly screening a dynamic effective region based on a projection relation and texture features, and removing static redundant scenes; performing sparse visual reasoning and depth regression only for the effective area to generate an initial depth map; carrying out physical structure correction on the depth map by utilizing geometric priori such as vertical upright posts and parallel top plates of the hydraulic support; and finally, constructing an optimization objective function containing deviation constraints of the vertical column perpendicularity and the top plate parallelism, and completing high-precision fusion of the incremental point cloud. According to the method, inference delay is reduced by reducing invalid calculation, structural distortion and registration drift caused by an underground severe environment are effectively inhibited by using a geometric law, and real-time and high-fidelity reconstruction of a working face scene is realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent coal mining technology, specifically to a lightweight three-dimensional reconstruction method and device for working faces that integrates geometric prior constraints. Background Technology

[0002] Currently, the construction of intelligent coal mines is in a stage of rapid development, and the transparency and visualization of fully mechanized mining faces are key supports for realizing unmanned mining. 3D reconstruction technology, as a bridge connecting the physical world and digital twins, provides remote monitoring personnel with an intuitive understanding of the operational scene by constructing a high-precision 3D model of the working face environment. This not only helps monitor the operating status of hydraulic supports and coal mining machines, but also provides necessary spatial reference data for the autonomous positioning, attitude determination, and path planning of the coal mining machine.

[0003] To address the 3D modeling requirements of fully mechanized mining faces, existing technologies primarily employ machine vision-based reconstruction schemes. Various explosion-proof cameras are deployed on the coal mining machine's body or supports, continuously acquiring image data of the working face as the machine moves. Visual algorithms extract features and calculate depth from the acquired image sequences, generating local 3D point cloud data. Subsequently, the system typically uses the Iterative Closest Point (ICP) algorithm for point cloud registration. This algorithm finds the nearest neighbor correspondence between the source and target point clouds, iteratively calculates the rotation and translation matrix, and thus gradually stitches the segmented local models into the global coordinate system, achieving incremental reconstruction of the narrow working face scene.

[0004] However, the underground working environment is unique, with high concentrations of dust and drastically changing lighting making it difficult to guarantee image quality. Traditional registration strategies that rely solely on minimizing geometric distance often fall short when dealing with low-quality or feature-deficient visual data. Existing methods ignore the inherent strong structural features of the working face itself, treating all points merely as indistinguishable geometric elements. Under noise interference, the algorithm is prone to converge to an incorrect pose solution, leading to a rapid amplification of accumulated errors. The generated panoramic models often exhibit deformations that violate physical laws, such as tilted hydraulic support columns and unnatural bending or fractures in the roof. This structural distortion deviates from physical reality and cannot meet the high-precision spatial decision-making requirements of intelligent mining.

[0005] Therefore, the present invention provides a lightweight three-dimensional reconstruction method and apparatus for working surfaces that integrates geometric prior constraints, in order to overcome the shortcomings of the prior art. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a lightweight 3D reconstruction method and device for working faces that integrates geometric prior constraints. This solves the problem that existing 3D reconstruction methods are prone to structural distortions that violate physical laws and have high computational redundancy, making it difficult to meet the high-performance real-time computing requirements of ground servers in the harsh environment of high dust and uneven lighting in underground coal mines.

[0007] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a lightweight three-dimensional reconstruction method for working surfaces that integrates geometric prior constraints, comprising the following steps: Step S1, Data Acquisition and System Initialization: Acquire the current frame RGB image and camera intrinsic parameters captured by the image acquisition unit, and read the global reference point cloud model constructed in the previous moment as the spatial reference reference for the current frame; Step S2, Dynamic Effective Region Filtering: Based on the camera intrinsic parameters, the pixels of the current frame RGB image are projected into three-dimensional space. The overlapping region and the newly added region are distinguished according to the spatial position relationship between the projection points and the global reference point cloud model. The dynamic overlapping region is determined by combining the texture change features in the overlapping region. The newly added region and the dynamic overlapping region are merged as the effective calculation region. Step S3, Lightweight Visual Model Inference: Sparse feature extraction and depth regression are performed only on image blocks within the effective computation area to generate local depth information; for areas outside the effective computation area, historical depth values ​​at corresponding positions in the global benchmark point cloud model are reused to generate an initial depth map. Step S4: Depth map optimization based on geometric prior: Based on the physical geometric characteristics that the hydraulic support column is perpendicular to the base plate and the top plate is parallel to the support top beam, identify the column region and the top plate region in the initial depth map, and perform vertical consistency correction on the depth sequence of the column region and parallelism correction on the depth value of the top plate region to generate an optimized depth map. Step S5, Incremental Point Cloud Fusion with Geometric Prior Enhancement: Convert the optimized depth map into a local point cloud, construct an optimization objective function that includes geometric structural deviation constraints, the geometric structural deviation constraints including at least column verticality deviation and top plate parallelism deviation; solve the optimization objective function to obtain the optimal pose, and fuse the local point cloud into the global reference point cloud model.

[0008] By adopting the above technical solution, this invention explicitly incorporates the inherent geometric structure of the coal mine working face into the entire process of 3D reconstruction. In the visual inference stage, by selecting dynamically effective regions and reusing historical depths, computational redundancy in the neural network is reduced, achieving lightweight inference. In the depth map generation and point cloud fusion stage, the geometric prior of the support columns being perpendicular to the roof and parallel to the roof is used to strongly constrain the visual calculation results, effectively solving the structural distortion and registration drift problems that are easily generated by conventional visual algorithms in the high-dust, low-texture environment underground, ensuring the consistency and reliability of the reconstructed model in terms of physical structure.

[0009] Preferably, the process of distinguishing between overlapping regions and newly added regions based on the spatial positional relationship between the projection point and the global reference point cloud model in step S2 includes: calculating the Euclidean distance between the projection point and the nearest neighbor point in the global reference point cloud model; if the Euclidean distance is less than or equal to a preset distance threshold, then the corresponding pixel is determined to be located in the overlapping region; if the Euclidean distance is greater than the distance threshold, then the corresponding pixel is determined to be located in the newly added region.

[0010] By adopting the above technical solution, the overlap range between the current field of view and the historical model is accurately defined using a three-dimensional spatial distance threshold, providing an accurate spatial reference for subsequent dynamic change detection.

[0011] Preferably, the process of determining the dynamic overlapping region in step S2 by combining the texture change features within the overlapping region includes: reading the frame image of the previous moment, calculating the local grayscale change rate of the pixels in the overlapping region of the current frame RGB image; the local grayscale change rate is obtained by calculating the average of the absolute values ​​of the difference between the grayscale values ​​of the current frame RGB image and the previous frame image at corresponding positions within a local neighborhood window centered on the pixel; if the local grayscale change rate is greater than a preset empirical threshold, then the corresponding pixel is determined to belong to the dynamic overlapping region.

[0012] By adopting the above technical solution, dynamic changes in the scene can be identified by monitoring photometric consistency. This can effectively capture scene updates caused by the movement of the coal mining machine or the peeling of the coal wall, while filtering out static backgrounds that have not changed, ensuring that computing resources are concentrated in high-value areas.

[0013] Preferably, the process of performing sparse feature extraction and depth regression only on image blocks within the effective computation area in step S3 includes: dividing the current frame RGB image into multiple non-overlapping image blocks, retaining only the image blocks located within the effective computation area to generate a sparse image block sequence; during the Transformer self-attention calculation process, establishing connections and calculating attention weights only between image blocks in the sparse image block sequence, and shielding computational interactions with image blocks outside the effective computation area; and performing dense depth decoding only on the image blocks in the sparse image block sequence.

[0014] By adopting the above technical solution, through sparse sampling at the input end and masking operation of the network's internal attention mechanism, the number of terms and computational complexity of the Transformer network are reduced. While ensuring the accuracy of dynamic region reconstruction, the algorithm's running efficiency is improved, making it adaptable to the real-time needs of ground servers with limited computing power of edge devices.

[0015] Preferably, the process of vertical consistency correction of the depth sequence of the pillar region in step S4 includes: extracting the pillar region in the initial depth map using a semantic segmentation mask; scanning the vertical direction of each column of pixel coordinates in the pillar region to extract the corresponding depth sequence; performing first-order polynomial fitting on the depth sequence to obtain the slope parameter; detecting the standard deviation of the fitting residual; if the standard deviation is greater than a preset threshold, determining that the depth data of this column is abnormal, using adjacent column data for interpolation repair, and forcibly constraining the slope parameter to zero.

[0016] By adopting the above technical solution, and utilizing the prior knowledge that the column is perpendicular to the base plate, the tilting or breakage of the column caused by visual matching noise in the depth map is forcibly corrected, ensuring that the reconstructed hydraulic support maintains the physical characteristic of being upright.

[0017] Preferably, the process of parallelism correction of the depth value of the roof area in step S4 includes: extracting the set of roof beam points belonging to the hydraulic support roof beam and the set of roof points belonging to the roadway roof; calculating the fitting plane normal vector of the set of roof beam points and the fitting plane normal vector of the set of roof points respectively; if the angle between the two fitting plane normal vectors is greater than a preset angle threshold, then fine-tuning the position of each point in the set of roof points along the direction of the fitting plane normal vector of the set of roof points.

[0018] By adopting the above technical solution and utilizing the parallel relationship between the roof slab and the roof beam, the non-physical bending or tilting that may occur in the roof slab reconstruction result is corrected, ensuring the flatness and rationality of the roadway roof structure.

[0019] Preferably, the process of constructing the geometric deviation constraint in step S5 includes: extracting the column point set from the local point cloud, fitting a spatial straight line and obtaining the column direction vector; extracting the top beam point set and top plate point set from the local point cloud, fitting a spatial plane and obtaining the top beam normal vector and top plate normal vector respectively; calculating the deviation of the angle between the column direction vector and the top beam normal vector relative to the vertical angle, as the column verticality deviation; calculating the angle between the top plate normal vector and the top beam normal vector, as the top plate parallelism deviation.

[0020] By adopting the above technical solution, the macroscopic structural features of the working surface are transformed into quantifiable geometric vector deviations, providing clear physical constraints for registration optimization.

[0021] Preferably, the optimization objective function in step S5 is composed of a weighted sum of the Euclidean geometric distance term, the column verticality deviation term, and the top plate parallelism deviation term; the Euclidean geometric distance term is the sum of squared point-to-point distances between corresponding points in the local point cloud and the global reference point cloud model; the process of solving the optimization objective function adopts the Newton gradient iteration algorithm.

[0022] By adopting the above technical solution, a structural deviation penalty term is added to the geometric distance minimization of the traditional iterative nearest point algorithm. This makes the pose solution process not only consider the fit of the point cloud surface, but also take into account the physical compliance of the overall structure. Thus, even when the point cloud quality is reduced due to high dust interference, robust incremental registration can still be achieved.

[0023] Preferably, the image block size in the sparse image block sequence is set to 14 by 14 pixels; the number of sparse image block sequences is less than the total number of image blocks after the whole image is divided.

[0024] By adopting the above technical solution, the granularity and scale of sparsification were clarified, ensuring the lightweight characteristics of the model input.

[0025] Secondly, the present invention provides a lightweight three-dimensional reconstruction device for working surfaces that integrates geometric prior constraints, comprising: The data acquisition module is configured to acquire the current frame RGB image and camera intrinsic parameters captured by the image acquisition unit, and read the global reference point cloud model constructed in the previous moment. The dynamic effective region filtering module is configured to project the pixels of the current frame RGB image onto a three-dimensional space based on the camera intrinsic parameters, distinguish overlapping regions from newly added regions according to the spatial positional relationship between the projection points and the global reference point cloud model, and determine the dynamic overlapping region by combining the texture change features in the overlapping region, and merge the newly added region and the dynamic overlapping region as the effective calculation region. The lightweight visual reasoning module is configured to perform sparse feature extraction and depth regression only on image blocks within the effective computation area to generate local depth information; for areas outside the effective computation area, the historical depth values ​​of the corresponding positions in the global reference point cloud model are reused to generate an initial depth map. The geometric prior depth optimization module is configured to identify the column region and the top plate region in the initial depth map based on the physical geometric characteristics that the hydraulic support column is perpendicular to the bottom plate and the top plate is parallel to the support top beam, and to perform vertical consistency correction on the depth sequence of the column region and parallelism correction on the depth value of the top plate region to generate an optimized depth map. The incremental point cloud fusion module is configured to convert the optimized depth map into a local point cloud, construct an optimization objective function that includes geometric structural deviation constraints, the geometric structural deviation constraints including at least column verticality deviation and top plate parallelism deviation; solve the optimization objective function to obtain the optimal pose, and fuse the local point cloud into the global reference point cloud model.

[0026] By adopting the above technical solution, the device realizes the entire process from data acquisition, region filtering, sparse inference to geometric constraint optimization through the logical division of hardware modules. It can be deployed on the ground server above the edge computing node in the coal mine to realize real-time, high-fidelity 3D reconstruction of the working face scene.

[0027] This invention provides a lightweight method and apparatus for 3D reconstruction of working surfaces that integrates geometric prior constraints. It has the following beneficial effects: 1. This invention introduces the verticality of the hydraulic support column and the parallelism of the roof as dual geometric constraints during the depth map optimization and point cloud fusion stages, explicitly embedding physical laws into the objective function of pose solving. This geometric prior constraint mechanism effectively overcomes the image feature matching failure problem caused by high dust and uneven lighting environments underground, suppresses scale drift and structural distortion in long-distance reconstruction, and maintains a high level of full-scene depth perception accuracy even in harsh environments, ensuring that the 3D model strictly conforms to the physical reality of the coal mine working face.

[0028] 2. This invention eliminates static redundant scenes through a dynamic effective region filtering mechanism and reduces the computational load of feature extraction and depth regression in invalid regions by combining the sparse input and sparse attention computation of a lightweight visual model. While ensuring the accuracy of dynamic region reconstruction, this method reduces the memory usage and inference latency of the ground server for edge computing devices, shrinks the effective feature extraction area per frame, and achieves smooth real-time 3D reconstruction, meeting the real-time perception requirements of rapidly moving coal mining machines.

[0029] 3. This invention constructs a joint optimization objective function that includes Euclidean geometric distance, column verticality deviation, and roof parallelism deviation, and uses a gradient iteration algorithm to solve for pose. Compared with traditional registration methods that rely solely on geometric distance, this invention ensures strict alignment of the local reconstructed model and the global reference model in terms of physical structure, effectively eliminating support tilting or roof bending caused by accumulated errors, solving the problem of structural fracture in the panoramic model, and providing highly reliable three-dimensional spatial data support for remote visualization monitoring and intelligent mining autonomous decision-making in underground operations. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the lightweight 3D reconstruction device for working surfaces that integrates geometric prior constraints according to the present invention. Figure 2 This is a flowchart of the lightweight 3D reconstruction method for working surfaces that incorporates geometric prior constraints, as described in this invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] See attached document Figure 1 This invention provides a lightweight 3D reconstruction device for a working face that integrates geometric prior constraints. The device includes an image acquisition unit, an edge computing processing unit, and a remote monitoring terminal. The image acquisition unit is installed on the coal mining machine body of the fully mechanized coal mining face and is configured to move with the coal mining machine and continuously acquire infrared or visible light video streams of the working face scene, covering the hydraulic supports, roof, and coal face areas. The image acquisition unit uses an explosion-proof industrial camera and is connected to the edge computing processing unit via a high-speed data transmission interface. The edge computing processing unit is deployed on the coal mining machine body or in a near-end explosion-proof control box and is configured to receive image data and execute 3D reconstruction algorithms. The remote monitoring terminal communicates with the edge computing processing unit via an underground fiber optic ring network or wireless communication network and is configured to receive and display the reconstructed panoramic 3D model of the working face.

[0033] The edge computing processing unit internally includes a memory and a processor. The memory stores the computer program, and the processor executes the computer program to implement the 3D reconstruction logic. From a functional module perspective, the device includes a data acquisition module, a dynamic effective region filtering module, a lightweight visual inference module, a geometric prior depth optimization module, and an incremental point cloud fusion module. The data acquisition module is configured to acquire the frame image and camera intrinsic parameters at the current moment, and read the global reference point cloud model constructed at the previous moment. This module is the input end of the entire system, ensuring that subsequent processing steps have the necessary visual data and spatial reference data. [Huang Zi 1] The dynamic effective region filtering module is connected to the data acquisition module and configured to receive the current frame image and the global reference point cloud model. This module projects the pixels of the current frame image back into 3D space based on camera intrinsic parameters, calculates the spatial distance between the projected points and the global reference point cloud model, and thus distinguishes between overlapping and newly added regions. For overlapping regions, this module further calculates the local grayscale change rate of the image and identifies regions with a change rate exceeding a preset threshold as dynamic overlapping regions. The dynamic effective region filtering module ultimately merges the newly added regions and dynamic overlapping regions into a single effective calculation region, thereby filtering out static redundant information in the scene that has not changed, reducing the computational load of subsequent steps.

[0034] The lightweight visual inference module is connected to the dynamic effective region selection module and configured to extract deep features from the effective computation region. This module divides the effective computation region into multiple non-overlapping sparse image patches and calculates attention weights only for these sparse image patches during feature extraction, shielding the computational interaction for invalid regions. In the depth decoding stage, the lightweight visual inference module performs dense depth regression only on image patches within the effective computation region, while for invalid regions, it directly reuses the historical depth values ​​at the corresponding locations in the global baseline point cloud model, ultimately generating an initial depth map containing depth information for the entire scene.

[0035] The geometric prior depth optimization module is connected to the lightweight visual inference module and configured to correct the initial depth map using the physical structural features of the coal mine working face. This module incorporates geometric rules governing the hydraulic support columns being perpendicular to the floor and the roof being parallel to the support beam. In the initial depth map, the module identifies the column region, eliminates tilt distortion by fitting a depth sequence and constraining its slope to near zero. Simultaneously, the module extracts point cloud data from the roof and beam, calculates the angle between their normal vectors, and fine-tunes the position of the roof point cloud based on parallelism deviation, outputting an optimized depth map that conforms to physical laws.

[0036] The incremental point cloud fusion module is connected to the geometric prior depth optimization module and configured to convert the optimized depth map into a local point cloud and fuse it into the global reference point cloud model. This module employs an improved iterative nearest-point algorithm, adding structural constraint terms to the traditional Euclidean distance minimization. Specifically, this module calculates the perpendicularity deviation between the column vector and the top beam normal vector, and the parallelism deviation between the top plate normal vector and the top beam normal vector in the local point cloud, incorporating these deviations as penalty terms into the pose optimization objective function. By minimizing this objective function, the incremental point cloud fusion module calculates the optimal rotation and translation matrix, accurately registering the local point cloud to the global coordinate system, thus completing the update of the panoramic model.

[0037] See attached document Figure 2 This invention provides a lightweight 3D reconstruction method for working surfaces that integrates geometric prior constraints. The method includes: Step S1, Data Acquisition and System Initialization. Acquire the current frame RGB image and camera intrinsic parameters captured by the image acquisition unit, and read the global reference point cloud model constructed in the previous moment as the spatial reference benchmark for the current frame.

[0038] Step S2, Dynamic Valid Region Filtering. Pixels in the current frame's RGB image are back-projected into 3D space along with the estimated depth. The spatial distance between the projected points and the global reference point cloud model is calculated to distinguish overlapping regions from newly added regions. For overlapping regions, the local grayscale change rate is calculated. Regions with change rates exceeding a threshold are identified as dynamically overlapping regions. The newly added regions and dynamically overlapping regions are merged and output as the valid calculation region.

[0039] Step S3, lightweight visual model inference. The effective computational region is divided into sparse image patches. During the Transformer self-attention computation, connections are established and attention weights are calculated only for these sparse image patches, shielding computational interactions in invalid regions. In the model output stage, dense depth regression is performed only on image patches within the effective computational region; for other regions, historical depth values ​​at corresponding locations in the global baseline point cloud model are directly reused to generate an initial depth map containing full-scene information.

[0040] Step S4: Depth map optimization based on geometric priors. Utilizing the physical geometric features of the hydraulic support columns being perpendicular to the base plate and the top plate being parallel to the support beam, the initial depth map is corrected pixel-by-pixel. The column regions in the initial depth map are identified and depth sequences are fitted. Anomalies that do not conform to the vertical distribution pattern are repaired through interpolation. Simultaneously, the top plate and top beam point sets are extracted, and the angle between their normal vectors is calculated. The depth values ​​of the top plate point set are fine-tuned based on the parallelism deviation, generating an optimized depth map that conforms to physical laws.

[0041] Step S5: Incremental point cloud fusion with geometric prior enhancement. The optimized depth map is converted into a local point cloud, and the column vectors, roof plate normal vectors, and roof beam normal vectors are extracted from the local point cloud. An optimization objective function is constructed, including Euclidean geometric distance, column verticality deviation, and roof plate parallelism deviation terms. The optimal rotation and translation matrix is ​​solved using a gradient iterative algorithm. The local point cloud is then accurately registered and fused into the global reference point cloud model, completing the 3D model update at the current moment.

[0042] See attached document Figure 2 Step S1 mainly performs data acquisition and system initialization operations, providing necessary input data and spatial reference for the subsequent 3D reconstruction process. This step first requires acquiring the camera's intrinsic parameter matrix and the current visual image data, and establishing the state of the global reference point cloud model.

[0043] The system first reads the pre-calibrated camera intrinsic parameters. The camera's internal parameters This describes the projection mapping relationship between points in three-dimensional space and pixels in a two-dimensional image, typically including parameters such as focal length, principal point coordinates, and distortion coefficients. Before system startup, the explosion-proof camera installed on the coal mining machine is calibrated using the Zhang Zhengyou calibration method or other precise calibration techniques, and the calculated camera intrinsic parameters are... It is stored in the memory of the edge computing processing unit for subsequent back projection calculations.

[0044] The image acquisition unit acquires infrared or visible light video streams from the working surface in real time and defines the frame image captured at the current moment as the current frame RGB image. The current frame RGB image It is a three-dimensional tensor, and its mathematical expression is as follows: ; in, Represents the height of the image; Represents the width of the image; This represents the red, green, and blue color channels. This image data... It is transmitted to the dynamic valid region filtering module for processing.

[0045] While acquiring image data, the system needs to load the global baseline point cloud model built in the previous moment. If the current time is the start time of the reconstruction process, the system will use the point cloud generated by the full reconstruction algorithm from the first frame image as the initial global reference point cloud model. If the current time is a subsequent time, the system reads the data from the memory from the previous time. The updated cumulative point cloud data serves as the current global baseline point cloud model. This global reference point cloud model contains the three-dimensional coordinate and color information of the reconstructed area of ​​the working surface, serving as a spatial reference system for incremental registration and fusion in the current frame.

[0046] See attached document Figure 2 Step S2 mainly performs a dynamic effective region filtering operation. This step aims to divide the current frame image into different semantic regions based on geometric projection relationships and photometric change features, and remove static redundant regions that have not changed, retaining only regions containing new environmental information or dynamic change information as input for subsequent inference.

[0047] The system first processes the RGB image of the current frame. Each pixel in Perform a 3D spatial back projection. Utilize the camera intrinsic parameters obtained in step S1. And the estimated depth value corresponding to the pixel, mapping the two-dimensional pixel coordinates to a three-dimensional spatial point. Subsequently, the system calculates the three-dimensional spatial point. With global benchmark point cloud model The Euclidean distance between the nearest neighbors. The system presets a distance threshold. In this embodiment, the distance threshold The value is set to 0.1 meters. If the calculated Euclidean distance is less than or equal to this distance threshold... If the calculated Euclidean distance is greater than the threshold value, then the pixel is determined to be located in the overlapping region, indicating that the geometry at that location already exists in the global model; If the pixel is located in the newly added area, it indicates that the location belongs to the blind area that has not been reconstructed.

[0048] For pixels identified as overlapping areas, the system further analyzes the changes in surface texture and photometric intensity to identify dynamic changes in the scene. The system reads the frame image from the previous moment. And calculate the current frame image. Local grayscale change rate of pixels in the overlapping region This local grayscale change rate reflects the visual feature differences of the same spatial location over a continuous time series. The calculation formula is as follows: ; in, Represented by pixels In this embodiment, a local neighborhood window centered on the local neighborhood is used. Set as a 7×7 pixel square neighborhood; This represents the total number of pixels within that neighborhood; and These represent the coordinates of the current frame image and the previous frame image, respectively. The grayscale value at that location.

[0049] After calculating the local grayscale change rate, the system compares it with a preset empirical threshold. Compare them. If the local grayscale change rate of a certain pixel... Greater than the empirical threshold If the pixel is found to belong to a dynamic overlap region, then the dynamic overlap region is determined to be within this region. The appearance of a dynamic overlap region corresponds to parallax changes caused by the movement of the coal mining machine or changes in scene structure caused by coal wall spalling. If the local grayscale change rate... Less than or equal to the empirical threshold If a pixel is identified as belonging to a static overlapping region, then the system determines that the pixel belongs to the static overlapping region. Finally, the system performs a region mask generation operation, logically merging all pixels marked as newly added regions with all pixels marked as dynamically overlapping regions to generate the final effective calculation region. This effective computing region It is the original image domain A subset of the region, where subsequent visual feature extraction and deep reasoning are performed only within that region.

[0050] See attached document Figure 2 Step S3 primarily performs lightweight DUSt3R model inference operations. This step aims to utilize the effective computational regions selected in step S2 to sparsify the input data and computational paths of the deep neural network, thereby reducing memory usage and improving inference speed. This step mainly includes three processing stages: Patch sparsity input construction, sparse attention mechanism computation, and locally dense depth decoding.

[0051] The system is first based on the effective computing region Construct a sparse input sequence. The system will extract the input sequence from the original image domain within the effective computational region. The pixel region within the image is divided into multiple non-overlapping image patches of 14×14 pixels. Only these patches located within the valid region are extracted to generate a sparse patch sequence. .in, Indicates the number of valid image blocks. This represents the total number of image patches after the entire image is divided. Since the effective computational region only covers newly added or dynamically changed parts of the scene, the number of effective image patches is much smaller than the total number of image patches in the entire image, i.e., it satisfies... The conditions. This sparse sampling strategy directly reduces the number of basic tokens entering the Transformer network.

[0052] After generating the sparse patch sequence, the system inputs it into the improved DUSt3R network and performs sparse attention mechanism computation. In the Self-Attention Layer of the Transformer module, the system only performs attention within the valid computation region. Connections are established between image patches, and the interaction of query vectors, key vectors, and value vectors is calculated, forcibly blocking the association calculation between valid and invalid image patches. Attention weight matrix The calculation formula is as follows: ; If image patch or Not part of the valid calculation area The corresponding attention weights Set it directly to 0. Among them, Indicates the first The query vector for each image patch; Indicates the first The key vector of each image patch, Indicates the dimension of the feature channel; This represents the matrix transpose operation; This represents the normalized exponential function. This mechanism ensures that computational resources are focused solely on feature extraction and correlation modeling of areas where the scene changes.

[0053] Finally, the system performs local dense depth decoding to generate an initial depth map. During the model output phase, the system only outputs data to the valid computational region. Dense depth regression prediction is performed on image patches within the region to calculate the new depth value for each pixel within that region. For images located within the effective computational region... For areas outside of this region, the system does not perform depth prediction, but instead directly indexes the global baseline point cloud model through projection relationships. The system reuses the depth values ​​calculated and stored at the corresponding location in the previous time step from the model. It then spatially stitches the newly predicted local depth values ​​with the reused historical depth values ​​to generate an initial depth map covering the entire field of view. This initial depth map serves as the basis for subsequent geometric prior optimization.

[0054] See attached document Figure 2 Step S4 primarily performs depth map optimization based on geometric priors. This step aims to utilize the inherent physical geometric features of the coal mine longwall face, namely the fact that the hydraulic support columns are perpendicular to the floor and the roof is parallel to the support beams, to optimize the initial depth map generated in step S3. Physical consistency correction is performed to eliminate structural distortions caused by visual matching errors. This step mainly includes two aspects: vertical consistency constraint treatment for the column area and parallelism regularization treatment of the top slab and top beam.

[0055] The system first performs vertical consistency constraint processing on the column region. The system utilizes a column mask generated by a semantic segmentation algorithm. Extract the pillar region from the initial depth map. Then, use the pillar mask... Within the coverage area, the system assigns coordinates to each column of pixels. Perform a vertical scan to extract the corresponding depth sequence. ,in and These represent the start and end coordinates of the pixels in the vertical direction, respectively. The system performs a first-order polynomial fitting on this depth sequence, and the fitting model is expressed as: ; in, Indicates the slope parameter; Indicates the intercept parameter; This represents the pixel coordinates in the vertical direction. After fitting, the system calculates the standard deviation of the fitting residuals. The system compares this standard deviation with a preset threshold, which is set to 0.03 meters in this embodiment. If the residual standard deviation... If the column's depth data is found to have abnormal noise or a matching error, the system determines that the column contains abnormal noise. For abnormal columns, the system uses data from the left and right adjacent reliable columns for interpolation repair and forces the slope parameter to be adjusted. The constraint is 0, that is This ensures that the corrected column depth structure remains consistent in the vertical direction, eliminating skew distortion.

[0056] Subsequently, the system performs parallelism regularization processing on the top slab and top beam. The system converts the depth data after column correction into a 3D point cloud and extracts the point set belonging to the hydraulic support top beam from it. and the set of points belonging to the roof of the tunnel The system uses the Random Sample Consensus Algorithm (RANSAC) to perform plane fitting on the two point sets respectively, to obtain the top beam plane. With top plate plane Top beam plane The equation is expressed as Top plate plane The equation is expressed as ,in Let be the normal vector of the top beam plane. Let be the normal vector of the top plate plane. The system calculates the angle between these two normal vectors. The calculation formula is as follows: ; The system determines the included angle. Whether it exceeds a preset angle threshold, which is set to 3 degrees in this embodiment. If This indicates a non-physical tilt angle deviation between the reconstructed top slab and the top beam. At this point, the system performs a point set analysis on the top slab. Each point in Along the top plate normal vector The position is fine-tuned in the direction. The coordinates of the point after fine-tuning are... The calculation formula is as follows: ; in, To adjust the step size coefficient, a value of 0.2 is used in this embodiment; The target parallel angle is usually set to 0. Through this iterative fine-tuning process, the system forces the top plate structure to gradually approach a state of parallelism with the hydraulic support top beam, and finally outputs an optimized depth map with a geometric structure that conforms to physical laws.

[0057] See attached document Figure 2 Step S5 primarily performs incremental point cloud fusion operations with geometric prior enhancement. This step aims to address the registration drift problem caused by feature loss in traditional iterative nearest-point algorithms in high-dust coal mine environments. By introducing the verticality of the hydraulic support column and the parallelism between the roof plate and roof beam as dual geometric constraints, high-precision physical alignment between the local point cloud and the global reference point cloud model is achieved. This step mainly includes three processing stages: local point cloud structure extraction, structural deviation quantification calculation, and constraint pose optimization solution.

[0058] The system first converts the optimized depth map output in step S4 into a 3D local point cloud. Based on the semantic segmentation results, the local point cloud is divided into pillar point sets. Top beam point collection and top plate point set For the set of column points The system uses the least squares method to fit a straight line in space. And extract the direction vector of the line. Geometric representation of the column. For the top beam point set. With top plate point set The system fits the spatial planes respectively and calculates the normal vector of the top beam plane. and the normal vector of the top plate plane .

[0059] After extracting the geometric feature vectors, the system calculates the geometric structural deviations to construct constraint terms. The system first calculates the column orientation vectors. With the normal vector of the top beam The angle between The calculation formula is as follows: ; Based on this included angle, the system defines the perpendicularity deviation. This is used to quantify the degree of geometrical orthogonality deviation between the column structure and the top beam structure in the current pose. Verticality deviation. The calculation formula is as follows: ; At the same time, the system calculates the top plate normal vector. With the normal vector of the top beam (i.e., the aforementioned) The angle between the two sides is used to define the parallelism deviation. This deviation is used to constrain the spatial parallelism between the top plate and the hydraulic support top beam. Parallelism deviation The calculation formula is as follows: ; Finally, the system constructs an ICP objective function that includes geometric constraints and solves for the optimal rotation matrix. And the translation vector. The objective function consists of the traditional point-to-point Euclidean distance term and the perpendicularity deviation term. And parallelism deviation term Together they form the core. The final optimization problem can be mathematically expressed as follows: ; in, Points in a local point cloud; For the corresponding point in the global baseline point cloud model; To determine the number of matching point pairs, the system uses the Newton-Raphson gradient algorithm to iteratively solve the objective function. During the iteration process, the gradient of the structural deviation term is backpropagated to the rotation matrix via the chain rule. The optimization direction is guided to simultaneously meet the requirements of geometric surface fit and physical structure rationality, thereby calculating the optimal pose parameters that can keep the panoramic model structure consistent and completing the accurate fusion of incremental point clouds.

Claims

1. A lightweight 3D reconstruction method for working surfaces that integrates geometric prior constraints, characterized in that, Includes the following steps: S1. Data Acquisition and System Initialization: Acquire the current frame RGB image and camera intrinsic parameters captured by the image acquisition unit, and read the global reference point cloud model constructed in the previous moment as the spatial reference reference for the current frame. S2. Dynamic effective region filtering: Based on the camera intrinsic parameters, the pixels of the current frame RGB image are projected into three-dimensional space, and overlapping regions and newly added regions are distinguished according to the spatial positional relationship between the projection points and the global reference point cloud model. The dynamic overlapping region is determined by combining the texture change features within the overlapping region, and the newly added region is merged with the dynamic overlapping region as the effective calculation region. S3. Lightweight visual model inference: Sparse feature extraction and depth regression are performed only on image blocks within the effective computation area to generate local depth information; For areas outside the effective calculation area, the historical depth values ​​of the corresponding locations in the global reference point cloud model are reused to generate an initial depth map. S4. Depth map optimization based on geometric prior: Based on the physical geometric characteristics that the hydraulic support column is perpendicular to the base plate and the top plate is parallel to the support top beam, the column region and the top plate region in the initial depth map are identified, and the depth sequence of the column region is corrected for vertical consistency, and the depth value of the top plate region is corrected for parallelism, so as to generate an optimized depth map. S5. Incremental point cloud fusion with geometric prior enhancement: The optimized depth map is converted into a local point cloud, and an optimization objective function containing geometric structural deviation constraints is constructed. The geometric structural deviation constraints include at least the verticality deviation of the column and the parallelism deviation of the top plate. The optimal pose is obtained by solving the optimization objective function, and the local point cloud is fused into the global reference point cloud model.

2. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 1, characterized in that, The process of distinguishing between overlapping and newly added regions based on the spatial relationship between the projection points and the global reference point cloud model in step S2 includes: Calculate the Euclidean distance between the projected point and the nearest neighbor point in the global reference point cloud model; If the Euclidean distance is less than or equal to a preset distance threshold, then the corresponding pixel is determined to be located in the overlapping region; If the Euclidean distance is greater than the distance threshold, then the corresponding pixel is determined to be located in the newly added region.

3. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 2, characterized in that, The process of determining the dynamic overlapping region in step S2 by combining the texture change features within the overlapping region includes: Read the frame image from the previous moment and calculate the local grayscale change rate of the pixels in the overlapping area of ​​the current frame RGB image; The local grayscale change rate is obtained by calculating the average of the absolute values ​​of the differences between the grayscale values ​​of the current frame RGB image and the previous frame image at corresponding positions within a local neighborhood window centered on the pixel. If the local grayscale change rate is greater than a preset empirical threshold, then the corresponding pixel is determined to belong to the dynamic overlapping region.

4. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 1, characterized in that, The process of performing sparse feature extraction and depth regression only on image patches within the effective computational region in step S3 includes: The current frame RGB image is divided into multiple non-overlapping image blocks, and only the image blocks located within the effective calculation area are retained to generate a sparse image block sequence; During the Transformer self-attention calculation process, connections are established and attention weights are calculated only between image patches in the sparse image patch sequence, shielding computational interactions with image patches outside the effective computation region; Dense depth decoding is performed only on image blocks in the sparse image block sequence.

5. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 1, characterized in that, The process of vertical consistency correction of the depth sequence of the column region in step S4 includes: The column region in the initial depth map is extracted using a semantic segmentation mask; The vertical direction of each column of pixel coordinates within the column area is scanned to extract the corresponding depth sequence; The slope parameter is obtained by fitting the depth sequence with a first-order polynomial. The standard deviation of the fitted residual is detected. If the standard deviation is greater than a preset threshold, the depth data of this column is determined to be abnormal. Interpolation is performed using data from adjacent columns to repair the abnormality, and the slope parameter is forcibly constrained to zero.

6. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 1, characterized in that, The process of parallelizing the depth value of the top plate region in step S4 includes: Extract the top beam point set belonging to the hydraulic support top beam and the top plate point set belonging to the roadway roof; Calculate the fitting plane normal vector of the top beam point set and the fitting plane normal vector of the top plate point set respectively; If the angle between the normal vectors of the two fitting planes is greater than a preset angle threshold, then the position of each point in the top plate point set is finely adjusted along the direction of the normal vector of the fitting plane of the top plate point set.

7. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 1, characterized in that, The process of constructing the geometric deviation constraint in step S5 includes: Extract the set of pillar points from the local point cloud, fit a spatial straight line, and obtain the pillar direction vector; Extract the top beam point set and top plate point set from the local point cloud, fit the spatial plane respectively, and obtain the top beam normal vector and top plate normal vector; The deviation of the angle between the column direction vector and the top beam normal vector relative to the vertical angle is calculated and used as the column verticality deviation. Calculate the angle between the normal vector of the top plate and the normal vector of the top beam, and use it as the parallelism deviation of the top plate.

8. The lightweight 3D reconstruction method for working surfaces integrating geometric prior constraints according to claim 7, characterized in that, The optimization objective function in step S5 is composed of a weighted sum of the Euclidean geometric distance term, the column verticality deviation term, and the top plate parallelism deviation term; The Euclidean geometric distance term is the sum of squared point-to-point distances between corresponding points in the local point cloud and the global reference point cloud model; The process of solving the optimization objective function adopts the Newton gradient iteration algorithm.

9. A lightweight three-dimensional reconstruction method for working surfaces incorporating geometric prior constraints according to claim 4, characterized in that, The size of the image blocks in the sparse image block sequence is set to 14 by 14 pixels; the number of sparse image block sequences is less than the total number of image blocks after the whole image is divided.

10. A lightweight three-dimensional reconstruction device for working surfaces that integrates geometric prior constraints, characterized in that, A lightweight 3D reconstruction method for working surfaces that integrates geometric prior constraints, as described in any one of claims 1-9, includes: The data acquisition module is configured to acquire the current frame RGB image and camera intrinsic parameters captured by the image acquisition unit, and read the global reference point cloud model constructed in the previous moment. The dynamic effective region filtering module is configured to project the pixels of the current frame RGB image onto a three-dimensional space based on the camera intrinsic parameters, distinguish overlapping regions from newly added regions according to the spatial positional relationship between the projection points and the global reference point cloud model, and determine the dynamic overlapping region by combining the texture change features in the overlapping region, and merge the newly added region and the dynamic overlapping region as the effective calculation region. The lightweight visual reasoning module is configured to perform sparse feature extraction and depth regression only on image blocks within the effective computation area to generate local depth information; for areas outside the effective computation area, the historical depth values ​​of the corresponding positions in the global reference point cloud model are reused to generate an initial depth map. The geometric prior depth optimization module is configured to identify the column region and the top plate region in the initial depth map based on the physical geometric characteristics that the hydraulic support column is perpendicular to the bottom plate and the top plate is parallel to the support top beam, and to perform vertical consistency correction on the depth sequence of the column region and parallelism correction on the depth value of the top plate region to generate an optimized depth map. The incremental point cloud fusion module is configured to convert the optimized depth map into a local point cloud, construct an optimization objective function that includes geometric structural deviation constraints, the geometric structural deviation constraints including at least column verticality deviation and top plate parallelism deviation; solve the optimization objective function to obtain the optimal pose, and fuse the local point cloud into the global reference point cloud model.

Images