Local update method and equipment for panoramic 3D dynamic reconstruction of coal mining machine follow-up working face

Abstract

This application relates to the fields of intelligent coal mining and computer vision technology, and discloses a method and equipment for local update of panoramic 3D dynamic reconstruction of a coal mining machine's follow-up working face. The method includes the following steps: First, global static initialization is performed to generate a global static reference model and calibrate the initial external parameter matrix; the position of the coal mining machine is acquired in real time, and update trigger detection is performed by calculating the spatial overlap rate between image features within the calculation window and the reference model. A trigger signal is generated only when the overlap rate is below a threshold. The cumulative displacement of the base is calculated using data from the hydraulic support stroke sensor, and a dynamic base compensation matrix is ​​constructed to adaptively calibrate the initial external parameters. A local point cloud is generated using the calibrated external parameters, and after registration, it is fused into the global model using a distance-weighted linear interpolation algorithm. This invention solves the coordinate drift problem caused by the displacement of the hydraulic support, ensures the geometric continuity of the model, reduces the computational load, and meets the real-time deployment requirements of ground servers.

Description

Technical Field

[0001] This invention relates to the fields of intelligent coal mining and computer vision technology, specifically to a method and equipment for local updating of panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face. Background Technology

[0002] With the rapid evolution of intelligent coal mining technology, constructing high-precision, high-time-efficiency three-dimensional digital twin models of fully mechanized mining faces has become a crucial foundation for realizing autonomous cutting, intelligent navigation planning, and disaster monitoring and early warning by coal mining machines. In the complex dynamic underground environment, real-time acquisition of panoramic three-dimensional spatial information of the working face is essential for ensuring the safety and efficiency of coal mining operations. Currently, underground three-dimensional spatial reconstruction mainly employs visual reconstruction technology based on multi-view images or point cloud modeling technology based on three-dimensional laser scanning.

[0003] However, existing mainstream visual reconstruction methods (such as COLMAP and OpenMVS) typically employ a global optimization strategy. Each model update requires backtracking through a sequence of nearly a hundred historical images, often taking several seconds or even minutes. This data latency lags far behind the actual moving cutting speed of the coal mining machine (0 to 10 meters per minute), making true real-time feedback impossible. Furthermore, these methods do not fully consider the operational characteristics of the coal mining machine as a mobile observation platform, tending to perform indiscriminate reconstruction calculations across the entire scene. This results in a significant waste of computing resources in static areas where environmental features remain unchanged, while in areas with drastic geometric changes near the drum, insufficient computing resources often lead to missed reconstructions, blurred textures, or geometric distortions, failing to meet the demands of real-time intelligent control.

[0004] On the other hand, while point cloud modeling methods based on 3D laser scanning have advantages in static geometric accuracy, their data acquisition typically relies on fixed sites or low-speed mobile platforms, making it difficult to continuously track and cover the working face in front of the high-speed moving coal mining machine. Furthermore, the laser scanning reconstruction process usually involves computationally intensive steps such as global point cloud denoising, full-scene meshing, and planar extraction, lacking an incremental update mechanism for local changes and failing to achieve dynamic response capabilities of simultaneous mining and reconstruction with updates. Especially under complex dynamic conditions caused by the continuous movement of hydraulic supports and the cutting action of the coal mining machine, traditional methods struggle to balance real-time performance with the consistency of spatial reference. Overcoming the resource bottlenecks and data lag caused by full-scale computation, and achieving dynamic, accurate local updates using limited ground server computing power, is a pressing technical challenge for the current 3D reconstruction of fully mechanized mining faces. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and equipment for local updates of panoramic three-dimensional dynamic reconstruction of coal mining face, which solves the problems of data lag and computing power bottleneck caused by camera base coordinate drift and full calculation in traditional visual reconstruction under the dynamic environment of hydraulic support pushing and coal mining machine rapid cutting.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a method and equipment for local update of panoramic three-dimensional dynamic reconstruction of a coal mining machine follow-up working face. The method includes global static initialization, follow-up window definition, update trigger detection, dynamic base adaptive calibration, and local incremental reconstruction and fusion steps.

[0007] During the global static initialization phase, the image acquisition devices within the fully mechanized mining face are controlled to synchronously acquire data, construct a global static benchmark model, and establish the initial external parameters of each acquisition device relative to the global coordinate system. To ensure the accuracy of the benchmark model, a weighted fusion strategy based on observation quality is adopted when processing data in the overlapping areas of the fields of view of adjacent acquisition devices. This strategy determines the fusion weights based on the angle between the line-of-sight vector of the observation point and the principal optical axis of the camera, as well as the distance from the observation point to the optical center, thereby reducing the impact of edge distortion on the model through geometric constraints.

[0008] During the dynamic window definition phase, the system no longer performs indiscriminate real-time calculations across the entire working face. Instead, it establishes a resource scheduling mechanism centered on the coal mining machine. By analyzing data from the coal mining machine's position detection sensors in real time, a dynamic update window centered on the current position of the coal mining machine is defined in the global coordinate system, and only image acquisition devices within the coverage area of ​​this window are activated.

[0009] During the update trigger detection phase, the system intelligently determines whether a time-consuming reconstruction task needs to be performed by calculating the spatial overlap rate. Specifically, the current image features are back-projected into 3D space, and the number of matching points whose distance to the nearest neighbor point on the surface of the historical benchmark model is less than a preset threshold is counted. The spatial overlap rate is obtained by calculating the ratio of the number of matching points to the total number of feature points. When the overlap rate is lower than the preset threshold, it indicates that a significant geometric change has occurred in the environment (such as the coal wall being cut). At this time, an update trigger signal is generated to avoid invalid calculations when the equipment is running idle or stopped.

[0010] In the adaptive calibration phase of the moving base, physical sensor constraints are introduced to address the base movement issue caused by the image acquisition device moving with the hydraulic support. In response to the update trigger signal, the system acquires the stroke data from the hydraulic support stroke detection sensor, calculates the difference between the current stroke value and the initial stroke value, and combines this with the direction coefficient of the pushing jack to determine the cumulative displacement of the image acquisition device base. Based on this physical displacement, a moving base compensation matrix including translational components is constructed to correct the initial extrinsic parameter matrix, generating the calibration extrinsic parameter matrix for the current moment. This step, prior to visual reconstruction, utilizes physical prior knowledge to eliminate most of the rigid displacement error.

[0011] In the local incremental reconstruction and fusion stage, the original local point cloud generated by the depth estimation model is transformed to the global coordinate system using the calibrated extrinsic parameter matrix, forming a coarsely registered point cloud. Subsequently, the overlapping area between the coarsely registered point cloud and the global baseline model is selected, and the optimal rigid body transformation matrix is ​​calculated using the iterative nearest-point algorithm. Fine registration is achieved by minimizing the sum of squared Euclidean distances between point pairs. Finally, to eliminate stitching artifacts between the locally updated region and the background model, a distance-weighted linear interpolation algorithm is used for fusion. This algorithm defines the core region radius and edge radius, constructing a weight field radiating outward from the center of the following window. New data is fully retained within the core region, weights decrease linearly within the transition region, and old data is retained outside the edge region. By weighted summing of the coordinates of the new and old data, the locally updated data is smoothly integrated into the global panoramic model.

[0012] A second aspect of the present invention provides a computer device.

[0013] The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0014] This invention provides a method and equipment for local updating of panoramic three-dimensional dynamic reconstruction of a coal mining machine's working face. It has the following beneficial effects: 1. This invention uses a dynamic base adaptive calibration step to correct the camera's extrinsic parameters in real time using the physical data of the hydraulic support stroke sensor. Combined with the definition of a follow-up window centered on the position of the coal mining machine, it achieves synchronous coupling between visual reconstruction and mechanical action. This mechanism eliminates the rigid displacement error caused by the frame shifting while ensuring that the local three-dimensional data is updated in real time along the cutting path of the coal mining machine, avoiding the data lag problem caused by the excessive time required for full-field reconstruction.

[0015] 2. This invention adopts an update trigger detection mechanism based on spatial overlap rate, combined with a spatial clipping strategy of the following window, which intelligently filters out invalid calculations in non-cutting states such as coal mining machine shutdown, idling, or maintenance. By starting the reconstruction process only when the environmental geometry changes substantially, the GPU computing load is reduced, thereby enabling smooth operation on the ground server terminal.

[0016] 3. This invention utilizes coarse and fine two-level cascaded registration and a distance-weighted linear interpolation fusion algorithm to ensure that the newly generated local point cloud can be smoothly and seamlessly integrated into the global static benchmark model. By constructing linear transition weights between the core area and the background area, the geometric discontinuities and seams caused by data splicing are eliminated, enabling the model update to accurately match the actual working condition changes, avoiding model collapse caused by accumulated errors, and improving the stability of the system's long-term continuous operation. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the logic flow of the follow-up window division and on-demand triggering mechanism of the present invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention provides a local update method for panoramic 3D dynamic reconstruction of a coal mining face using a follow-up mining machine. This method operates in a specific hardware environment of a fully mechanized mining face. This hardware environment includes a hydraulic support group, a scraper conveyor, a coal mining machine, an image acquisition device, sensor components, and a data processing unit.

[0020] The hydraulic support group consists of multiple hydraulic supports arranged linearly along the dip of the longwall mining face. Each hydraulic support includes a top beam, a base, a column, and a moving jack. The moving jack connects the hydraulic support base to the scraper conveyor and is used to perform the pushing and moving actions. The scraper conveyor is laid under the front end of the hydraulic support group. The coal mining machine straddles the scraper conveyor and can perform reciprocating cutting operations along the scraper conveyor track.

[0021] The image acquisition device group comprises several image acquisition devices. These devices are mounted on the lower front end of the top beam of the hydraulic support. In this embodiment, the image acquisition devices are not installed on every hydraulic support, but rather deployed at predetermined intervals. For example, one image acquisition device is installed every two hydraulic supports. The optical axis of the image acquisition device's lens is set to face the coal wall and tilt downwards. In this embodiment, the pitch angle of the image acquisition device is set to -20 degrees. This installation angle ensures that the field of view of the image acquisition device covers the front beam of the hydraulic support, the side guard plate, the scraper conveyor cable trough, and the coal wall area.

[0022] The installation spacing between two adjacent image acquisition devices is matched to the field of view of the image acquisition devices. The field of view and installation height of the image acquisition devices are configured so that the images acquired by the two adjacent image acquisition devices have a spatial overlap area. In this embodiment, the field of view overlap rate of adjacent image acquisition devices is set to be greater than or equal to 30%. This overlap rate index is used to ensure sufficient feature matching points between consecutive image sequences to support subsequent 3D reconstruction and stitching processing. The image acquisition devices can be depth cameras or binocular stereo cameras, used to acquire RGB image data or depth image data of the work surface scene.

[0023] The sensor assembly includes a coal mining machine position detection sensor and a support travel detection sensor. The coal mining machine position detection sensor, specifically a travel encoder, is mounted on the traveling mechanism or body of the coal mining machine. This sensor measures the absolute position coordinates of the coal mining machine in the direction of its advance along the working face in real time. These coordinates are used to determine the real-time position of the coal mining machine within the working face.

[0024] The support stroke detection sensor is installed inside or outside the pushing jack of each hydraulic support. Specifically, the support stroke detection sensor is a magnetostrictive displacement sensor or a linear displacement sensor. The support stroke detection sensor is used to measure the extension and retraction of the pushing jack in real time. This extension and retraction characterizes the relative displacement change of the hydraulic support relative to the scraper conveyor. When the hydraulic support performs a moving action, the data output by the support stroke detection sensor reflects the displacement of the hydraulic support top beam and the image acquisition device mounted on it relative to the global coordinate system of the working surface.

[0025] The data processing unit is connected to the image acquisition device group and sensor components via an industrial Ethernet or fiber optic network. The data processing unit is deployed in an explosion-proof computer underground or a ground server. It receives image stream data from the image acquisition device sequence, position data from the coal mining machine position detection sensor, and displacement data from the support stroke detection sensor. The data processing unit is equipped with a parallel computing processor to execute subsequent 3D reconstruction algorithms, pose calibration algorithms, and point cloud fusion algorithms.

[0026] In summary, this embodiment constructs a physical system that incorporates visual perception and motion state perception. Through fixed physical connections and a defined data transmission path, this system provides the hardware foundation and data support for resolving the problem of base coordinate system variations caused by hydraulic support movement.

[0027] See attached document Figure 1 The method provided by this invention is executed by a data processing unit and aims to coordinate multi-source heterogeneous data to achieve incremental updates of local scenes while ensuring geometric accuracy. The method mainly includes a global static initialization phase and a dynamic local update phase.

[0028] During the global static initialization phase, the data processing unit controls the image acquisition device group to perform full-coverage imaging of the fully mechanized mining face. The data processing unit receives the initial frame image sequence acquired simultaneously by all image acquisition devices. It then calls a pre-set depth estimation model or stereo matching algorithm to process the initial frame image sequence in parallel, generating corresponding depth map data. Subsequently, using the calibration intrinsic parameters of each image acquisition device, the depth map is back-projected into 3D space, generating initial sparse or dense point clouds from multiple perspectives. Based on the preset installation positions and attitude information of each image acquisition device in the world coordinate system, the data processing unit uniformly transforms the multi-view point clouds to the global coordinate system. For point cloud data in overlapping fields of view of adjacent image acquisition devices, a weighted fusion operation is performed to eliminate redundant data and smooth noise. Finally, through voxelized mesh processing and surface reconstruction algorithms, a global static reference model of the working face is generated. This reference model serves as the geometric reference frame for subsequent dynamic updates.

[0029] After entering the follow-up local dynamic update stage, the method enters a cyclic monitoring and processing flow. The data processing unit continuously reads the real-time position data uploaded by the coal mining machine's position detection sensor at a set frequency (e.g., 20Hz). Based on this real-time position data, the data processing unit defines a follow-up update window in the global coordinate system. This follow-up update window is a spatial range extending forward and backward along the working face with the current geometric center of the coal mining machine as the origin. The data processing unit only activates or calls the data stream of the image acquisition device within the coverage area of ​​this follow-up update window, and does not process or only caches the image data outside the window, thereby realizing the regional allocation of computing resources.

[0030] After determining the follow-up update window, the update trigger detection step is executed. The data processing unit extracts the feature point set of the image at the current moment within the follow-up update window and projects this feature point set onto the local model or global baseline model generated at the previous moment. The spatial overlap rate between the current feature point set and the historical model surface point set is calculated. The calculated spatial overlap rate is compared with a preset threshold. When the spatial overlap rate is higher than or equal to the preset threshold, it is determined that the coal wall morphology in the current field of view has not changed significantly (e.g., the coal mining machine is in an unloaded traction state), and the process skips the reconstruction step, maintaining the current model state. When the spatial overlap rate is lower than the preset threshold, it is determined that the environmental geometry in the current field of view has changed (e.g., the coal mining machine has performed a cutting operation, resulting in the exposure of new coal wall), and the system triggers the local incremental reconstruction process.

[0031] After triggering the local incremental reconstruction process, the adaptive calibration step for the moving base is executed first. The data processing unit reads the support stroke detection sensor data of the hydraulic support located within the follow-up update window. By analyzing the extension and retraction of the jacks, the displacement vector of each image acquisition device base relative to the initial moment is calculated. The data processing unit constructs a dynamic compensation matrix and uses this matrix to correct the external parameter matrix of the image acquisition device in real time, obtaining the calibrated camera pose at the current moment. This step eliminates the imaging coordinate system drift caused by the support movement.

[0032] Subsequently, the local point cloud generation and registration steps are performed. Using the calibrated camera pose and image data within the current time window, local point cloud data reflecting the current state of the coal face is generated. The data processing unit first coarsely transforms the local point cloud to the global coordinate system based on the position information of the coal mining machine. Then, using a point cloud registration algorithm (such as the iterative nearest point algorithm), and constrained by unchanged areas in the global reference model (such as the features of the hydraulic support top beam), the local point cloud is finely registered, and the optimal rigid body transformation matrix is ​​calculated to ensure that the local point cloud is geometrically strictly aligned with the global environment.

[0033] Finally, boundary smoothing and model updating steps are performed. The data processing unit identifies the spatial boundary region between the locally updated point cloud and the original global model. Within the boundary region, a linear transition field based on distance weights is constructed. For voxels or mesh vertices within the boundary region, the fusion weight of the old and new data is calculated based on their distance from the center of the update window. This weight is used to interpolate and fuse the newly generated local point cloud data with the original global model data, generating a smooth updated surface. The data processing unit uses the fused local data to replace the corresponding spatial region in the global model, completing one dynamic update cycle, and outputs the updated panoramic 3D model for visualization or subsequent intelligent control decisions.

[0034] This embodiment details the process of constructing a high-precision three-dimensional reference model of the entire working face using multi-view image data before the start of coal mining operations or during the system reset phase. This process mainly includes four core steps: synchronous image acquisition, depth map generation and backprojection, weighted fusion of multi-view point clouds, and surface mesh reconstruction.

[0035] Step 1: Synchronous acquisition of images across the entire field.

[0036] The data processing unit sends a synchronization trigger signal to all image acquisition devices deployed on top of the hydraulic support. The image acquisition devices simultaneously capture a sequence of panoramic images of the longwall mining face. ,in This represents the total number of image acquisition devices. This step ensures that all acquired images reflect the working surface state at the same static moment, avoiding time deviations caused by non-rigid movement of the equipment. The acquired image data is transmitted to the ground server via gigabit Ethernet or fiber optic cable.

[0037] Step 2: Depth map generation and spatial back projection based on deep learning.

[0038] After receiving the image sequence, the data processing unit uses a preset stereo matching depth estimation model to process each frame of the image. The model employs a dense stereo matching network based on the Transformer architecture (e.g., the DUSt3R architecture), which has the ability to output absolute depth values ​​without explicit epipolar correction.

[0039] For the The image from the perspective of the image acquisition device is used to output the corresponding depth map from the model. Each pixel value in the depth map This represents the Euclidean distance or vertical depth (Z-depth) from the pixel to the camera's optical center. Subsequently, the intrinsic parameter matrices pre-calibrated by each image acquisition device are used... The pixel coordinates of the two-dimensional image plane are back-projected onto the three-dimensional camera coordinate system.

[0040] Specifically, for any pixel point on the image plane Its three-dimensional coordinates in the camera coordinate system The calculation formula is as follows: ; in, This represents the inverse of the camera intrinsic parameter matrix. Let be the homogeneous coordinate vector of the pixel. Using this formula, the system converts a two-dimensional depth map into massive amounts of three-dimensional discrete point data. Step 3: Weighted fusion of multi-view point clouds based on observation quality.

[0041] Because of the overlap between adjacent image acquisition devices, the same physical point can be observed simultaneously by multiple cameras. To address the redundancy problem of multi-source data in overlapping areas and reduce geometric errors caused by camera lens edge distortion, this embodiment employs a weighted fusion strategy based on observation geometric quality.

[0042] The data processing unit processes all data according to the mounting pose of each camera. Unified transformation to world coordinate system This forms a global point cloud set. For points within overlapping regions, their fusion weights are calculated. This weight is primarily determined by the angle between the observation line of sight and the optical axis. and observation distance Decision. The specific calculation formula is as follows: ; in, Indicates the first The angle between the line-of-sight vector at each observation point and the principal optical axis of the camera. The smaller the value, the closer the point is to the image center, the less lens distortion, the higher the observation quality, and the greater the weight. This represents the Euclidean distance from the point to the camera's optical center. In depth estimation, closer distances generally result in higher measurement accuracy; therefore, the weight is inversely proportional to the distance. A small positive constant (e.g., 1 × 10⁻⁶) is set to prevent the denominator from being zero. -5 ).

[0043] Based on the above weights, the spatial points in the overlapping area are weighted and averaged to calculate the position, thereby generating a denoised high-confidence fused point cloud.

[0044] Step 4: Point cloud optimization and surface mesh reconstruction.

[0045] Although the fused point cloud data possesses a global geometric structure, it may still contain outlier noise points. The data processing unit employs a statistical outlier filtering method. For each point in the point cloud, it evaluates the local dispersion by obtaining the average distance between the point and a specified number of nearest neighbors. Then, it removes outlier noise points that exceed the threshold by comparing the average distance with a judgment threshold determined based on the overall distance distribution characteristics.

[0046] Subsequently, the optimized point cloud is subjected to voxel downsampling. The side length resolution of the voxel mesh is set (e.g., 2 cm), and only one centroid point is retained within each voxel to achieve uniform reduction of data volume. Finally, the downsampled point cloud and its normal vector information are processed using the Poisson Surface Reconstruction algorithm to construct a closed triangular mesh model. This triangular mesh model serves as the global static baseline model. It will be stored in video memory as the geometric base map and registration reference for the subsequent local dynamic update stage.

[0047] This embodiment details how, during the advancement of a fully mechanized mining face, the impact of dynamic base displacement on the visual reconstruction system is quantified by establishing a mapping relationship between the kinematic model of the support and the sensor data stream.

[0048] Analysis of the impact of the camera movement on the camera coordinate system: In the longwall mining face, the image acquisition device is rigidly fixed to the front end of the top beam of the hydraulic support. Therefore, the image acquisition device and the top beam of the hydraulic support constitute a rigid motion unit. The system's global coordinate system (WorldCoordinate System)... ) is usually defined at the initial cut position of the working face or at a certain static reference point, where The shaft inclines along the working surface (in the direction of the scraper conveyor). The shaft runs along the working face (in the direction of propulsion). The axis is vertically upward.

[0049] When the hydraulic support performs the moving action, the base of the hydraulic support remains stationary relative to the scraper conveyor (at this time, the scraper conveyor acts as a fulcrum). The pushing jack extends, pushing the top beam and column of the hydraulic support along... A translational motion occurs in the positive direction of the axis. Let the first... An image acquisition device at the initial moment The position of the optical center in the global coordinate system is At that moment If the support structure moves, the resulting physical displacement vector is: Due to the rigid connection, the optical center position of the image acquisition device changes synchronously. .

[0050] If the visual reconstruction algorithm ignores this physical displacement and still uses the camera extrinsic matrix calibrated at the initial time for spatial point back projection, the newly generated point cloud data will produce a result in a different coordinate system compared to the original coordinate system. Systematic deviations of opposite direction and equal magnitude. In three-dimensional space, this deviation manifests as layering or ghosting of point clouds, meaning that the same coal face feature point cannot overlap in the two reconstructions before and after the support shift. Therefore, the mechanical kinematic parameters of the hydraulic support must be incorporated into the coordinate transformation chain of the vision system, treating the camera coordinate system as a time-varying coordinate system. Changing dynamic coordinate system .

[0051] Sensor data reading and displacement analysis: In order to obtain the above physical displacement vector in real time The data processing unit communicates with the electro-hydraulic control system of the hydraulic support via an industrial bus (such as CAN bus or Ethernet) to read the stroke detection sensor data installed in the push jack.

[0052] Data collection mechanism: The data processing unit polls the sensor status of all hydraulic supports within the follow-up update window at a preset sampling frequency (e.g., 50Hz). The collected data packets include the support number (ID), the current timestamp (t), and the extension rod stroke value of the jack. This stroke value is typically output directly from a magnetostrictive displacement sensor, and the unit is millimeters (mm).

[0053] Data cleaning and status determination: Due to the complex electromagnetic environment downhole and the minute fluctuations in the hydraulic system, the original stroke value... High-frequency noise may be present. The data processing unit first performs a moving average filter on the raw data. Then, a dead-zone threshold is set. (e.g., 10mm).

[0054] like The system determines whether the stent is in a static or slightly moving state, ignoring the minute change.

[0055] like The system determines that the support is performing a moving action and enters the displacement calculation process.

[0056] Calculation of relative cumulative shift: This method focuses on the displacement change of the camera relative to the global initialization moment. Therefore, at the instant the global initialization is completed (moment 1), the system... Record the first The initial stroke value of the hydraulic support jack of the image acquisition device is denoted as . .

[0057] At any subsequent time Read the current filtered stroke value At this time, the first Image acquisition device No. 1 moves along the advancing direction ( The cumulative displacement of the axis The calculation is as follows: ; in, This is the direction coefficient. Due to different installation methods for the jacks (orthogonal or reverse installation), an increase in the sensor value may indicate that the support has moved forward or backward. The system determines this based on the preset hydraulic system installation configuration file. The value (+1 or -1) ensures Always characterize the support along the global coordinate system Physical displacement in the positive direction of the axis.

[0058] Calculated The data is stored in the system's dynamic state register and serves as the direct input parameter for constructing the dynamic base compensation matrix, used in subsequent steps to correct the translation vector of the camera's extrinsic parameter matrix. Through this analytical process, the system transforms the one-dimensional linear sensor data from the mechanical side into three-dimensional spatial displacement constraints from the visual side.

[0059] This embodiment continues the previous discussion of sensor displacement data ( The analysis of the physical constraint is explained in detail, and how to use the physical constraint to construct a mathematical model and correct the spatial mapping relationship of the visual system is described.

[0060] Construction of the dynamic base compensation matrix: The data processing unit obtains the first The cumulative displacement of the hydraulic support of the image acquisition device relative to the initial moment. Subsequently, a dynamic base compensation matrix was constructed based on the principles of rigid body kinematics. In the global coordinate system defined in this embodiment, it is assumed that the movement path of the hydraulic support strictly follows the global coordinate system. The movement is carried out along the axis (working face direction), and the minor pitch and roll vibrations during the frame shifting process are ignored (or it is assumed that these vibrations are further eliminated during the local registration stage).

[0061] Based on this geometric constraint, the compensation matrix is ​​defined as a matrix describing the distance along the line. A 4×4 homogeneous transformation matrix for axis translation. This matrix is ​​only valid for the translation vector. The components contain variables, and the rotated submatrix remains an identity matrix. Its specific mathematical expression is as follows: ; Among them, the elements on the main diagonal This indicates that no transformation was introduced in the rotation dimension; the first and third lines indicate that... and In the case where there is no displacement on the axis, the second row Indicates in The displacements on the axes, the fourth row is the standard form of homogeneous coordinates, ensuring that this is a valid homogeneous transformation matrix (typically used for operations such as translation, rotation, and scaling). This matrix represents the physical fact that the current position of the camera in the base space, relative to the base space position at system initialization, is... A distance of 1000 kJ / m² occurred in the axial direction. Translation.

[0062] Real-time correction of camera extrinsic parameters: During the system global initialization phase, the data processing unit has been calibrated and stored the first... The initial external parameter matrix of the image acquisition device at a static moment relative to the global origin is denoted as follows: This matrix describes the transformation relationship from the camera coordinate system to the world coordinate system at the initial moment.

[0063] Before processing each frame in the local dynamic update cycle, the data processing unit uses the constructed compensation matrix. The initial extrinsic parameters are updated by left multiplication. Since the compensation matrix describes the motion of the base in the world coordinate system, according to the coordinate transformation rules in linear algebra, the corrected current-time camera extrinsic parameter matrix is ​​obtained. The calculation formula is as follows: ; Through this matrix multiplication operation, the system injects the measurement results of the physical sensors into the visual imaging model. It contains the latest location information; in essence, it places the camera's virtual optical center along the world coordinate system. The axis was translated The distance. If the system is also connected to a tilt sensor for detecting the bracket's attitude (such as the base tilt angle), then The rotation submatrix (the first 3×3 part) in the matrix can also be replaced by a rotation matrix based on Euler angles, thereby achieving full-range compensation of six degrees of freedom.

[0064] Application of modified extrinsic parameters in coordinate transformation: Obtain the corrected extrinsic parameter matrix Then, the data processing unit applies it to the spatial mapping process of the local point cloud.

[0065] First, the image acquisition device acquires a local image at the current moment, and then generates a set of local discrete point clouds in the current camera coordinate system using a depth estimation model. Any point in this set It only indicates the relative position of the point with respect to the current optical center of the camera.

[0066] To stitch this local point cloud into the global model, it must be transformed to the world coordinate system. The data processing unit performs the following rigid body transformation using the corrected extrinsic matrix: ; in, These are the absolute coordinates of the point in the global world coordinate system.

[0067] Through the above applications, even if the image acquisition device moves physically by several meters along with the hydraulic support, as long as... The measurement is accurate, and the result is calculated using this formula. It can still accurately fall on the actual spatial position of the coal wall in the global coordinate system. This step eliminates the drift of the reconstructed point cloud caused by the movement of the base from a geometric perspective, ensuring that the initial values ​​of the local point cloud and the global background model are within a reasonable convergence region when performing fine registration (ICP) in subsequent steps, thus avoiding the registration algorithm from getting trapped in local minima or diverging.

[0068] See attached document Figure 2 This embodiment describes in detail how the system dynamically defines the calculation range based on the real-time position of the coal mining machine, and uses a feature-based spatial overlap rate algorithm to intelligently determine whether the current scene needs to be updated by three-dimensional reconstruction.

[0069] Dynamic definition of the window that updates dynamically: The data processing unit receives the real-time position coordinates of the coal mining machine from the position detection sensor via the industrial bus. This coordinate represents the orientation of the cutting drum of the coal mining machine at the working face. The physical center position on the axis. Based on this position, the data processing unit constructs a virtual, dynamically updated window in the global coordinate system. .

[0070] Update window The coverage area is defined as .in, This is the forward sight distance (e.g., 7.5 meters) in the direction the coal mining machine is traveling. The backward coverage distance is set to 7.5 meters. The total length of the window (e.g., 15 meters) is set to completely cover the length of the coal mining machine and the coal wall area just cut by the front and rear drums of the coal mining machine. The data processing unit iterates through the list of installation locations of the image acquisition device group and places the installation coordinates in the window. Image acquisition devices within the designated area are marked as active. Only video streams uploaded by active image acquisition devices are sent to video memory for further processing, while data from image acquisition devices outside the window are only locally cached or transmitted infrequently. This geofencing-based resource scheduling strategy strictly limits high-performance 3D reconstruction operations to the core working area of ​​the work surface.

[0071] Trigger detection based on spatial feature overlap rate: After identifying the active image stream, the data processing unit does not immediately perform time-consuming dense reconstruction on each frame of the image, but instead first performs lightweight trigger detection.

[0072] Feature extraction and spatial projection: For moving windows Current moment For each image frame, the data processing unit calls a feature extraction algorithm (such as SIFT or ORB) to extract key feature points from the image. Then, it combines the fast depth estimation result (or sparse depth sampling) of the current frame with the corrected camera extrinsic parameters. These two-dimensional feature points are back-projected into three-dimensional space to form the feature point set of the current frame. .

[0073] Simultaneously, the data processing unit indexes the historical model data (i.e., the global baseline model or the model after the last update) corresponding to the spatial region from the video memory, extracts the surface point data within the region, and forms the old model point set. .

[0074] Spatial overlap rate calculation: The system calculates the current feature point set Point set of the old model The degree of geometric overlap between elements is used to quantify environmental changes. Spatial overlap rate is defined. The calculation formula is as follows: ; in, For the current frame feature point set any point in, For the old model point set any point in, Point With point The Euclidean distance between them Point The shortest distance to the old model point set. The distance threshold for judgment (e.g., 5 cm). If the shortest distance is less than... Then the decision point It coincides with a point in the old model, meaning the environmental characteristics at that location have not changed. This represents the number of elements in the set. The numerator of the formula calculates the number of matching points, and the denominator is the total number of feature points in the current frame.

[0075] Triggering judgment logic: The data processing unit will calculate the overlap rate With preset trigger threshold (In this embodiment, the value is set to 0.7, i.e., 70%) for comparison: Non-triggered case: If This indicates that most feature points (over 70%) within the current field of view can be found in the old model. This typically corresponds to situations where the coal mining machine is running idle (without cutting the coal face), undergoing maintenance, or repeatedly passing through a cut area. In this case, the system determines that the environmental geometry has not changed significantly, terminates the subsequent reconstruction process of the current frame, and directly reuses the old model data for display, thereby saving computational resources.

[0076] Triggering conditions: If This indicates that there are a large number of new feature points in the current field of view that cannot be matched in the old model. This corresponds to the fact that the coal mining machine drum has just cut the coal wall, causing the old coal wall surface to disappear and exposing the new coal wall surface. At this time, the system determines that a substantial geometric update has occurred in the environment, immediately generates a trigger signal, and activates the subsequent local incremental reconstruction and fusion module.

[0077] This embodiment details how to generate high-precision local 3D data and seamlessly integrate it into the existing global baseline model after the system determines that the environment has changed, so as to achieve real-time dynamic updates of the panoramic model of the working surface. The process mainly includes three steps: local dense reconstruction, two-level cascaded registration, and boundary smoothing fusion based on linear weights.

[0078] Generation of locally dense point clouds: Once the trigger mechanism generates an update command, the data processing unit immediately locks the follow-up update window. The image acquisition device is in an active state. The data processing unit obtains the current time. The image frames are processed, and a depth estimation model (such as a Transformer-based stereo matching network) is invoked again to perform full-resolution depth inference on the images within the window.

[0079] Unlike the trigger detection phase, which uses only sparse feature points, this step aims to recover the geometric details of the scene. After the model outputs a high-resolution depth map, each valid pixel in the depth map is back-projected to the current camera coordinate system space using the camera intrinsic parameter matrix, generating a locally dense point cloud set. This point cloud collection contains the latest coal facet surface, cut marks, and current status information of the hydraulic supports within the current field of view.

[0080] Two-level cascaded registration (coarse registration and fine registration): In order to transform the newly generated local point cloud To ensure accurate placement in the global coordinate system, this embodiment employs a two-level strategy combining sensor-guided coarse registration and geometrically constrained fine registration.

[0081] Coarse registration based on moving base compensation: First, the camera extrinsic parameter matrix calibrated by the moving base is used. , local point cloud Transforming from the camera coordinate system to the world coordinate system yields a coarsely aligned point cloud. : ; This step utilizes measurement data from physical sensors to eliminate most of the rigid displacement errors caused by the hydraulic support's movement, thus... The position in the global space is already very close to the actual position, providing a good initial solution for subsequent fine registration.

[0082] Fine registration based on the ICP algorithm: Because sensor data may contain slight drift or installation errors, Compared with the global benchmark model Millimeter-level misalignment may still exist in the overlapping areas. To address this, the data processing unit employs the Iterative Closest Point (ICP) algorithm for fine-grained alignment.

[0083] System selection Compared with the global benchmark model The spatially overlapping region is used as the registration region. Within this region, the system seeks structures with relatively stable geometric features and no non-rigid deformation (such as the top beam of an adjacent unmoved support, the edge of a side panel, or the boundary of an uncut coal wall) as anchor points.

[0084] Construct an optimization objective function and solve for a fine-tuning transformation matrix through iterative calculation. This makes the points in the local point cloud The nearest neighbor in the global model Minimize the sum of squared Euclidean distances between them: ; in, The number of corresponding point pairs participating in the registration. For rough matching of accurate point clouds The first in One point, For the global model and The nearest corresponding point Represents the transformation matrix Optimize to minimize the objective function. Let be the rigid body transformation matrix to be solved (containing small rotation and translation components).

[0085] By solving this optimization problem, the final precise point cloud is obtained. This ensures a high degree of consistency in the geometric structure between the old and new data.

[0086] Boundary linear weighted fusion: Definition of fused weight field: The finely registered local point cloud Directly overlaying onto the global model may create geometric discontinuities or seams at the stitching boundaries. To ensure the visual continuity and geometric smoothness of the panoramic model, this embodiment employs a linear weighted fusion algorithm based on the distance field.

[0087] The data processing unit defines the geometric center of the dynamically updated window as... For any spatial point within the local point cloud coverage area. Define its fusion weights The weight is about the point. Distance to the center of the window The linear decay function.

[0088] Let the effective radius of the window be... The weight calculation formula is as follows: ; in, The core region radius (the region with a weight of 1) is where all new data is used. The edge radius is defined; data exceeding this range has a weight reduced to 0, and the old data is fully retained. and The transition zone between them has a linearly decreasing weight.

[0089] Point cloud fusion update for each voxel or mesh vertex within the transition zone, its final spatial coordinates The newly generated local point cloud coordinates Coordinates in the original global model Joint decision: ; Through this weighted fusion operation, the coal face morphology at the center of the updated region is completely updated to the current cut state, while the edges of the updated region smoothly transition into the surrounding historical background, eliminating splicing marks. Finally, the data processing unit writes the fused data block into a global octree or voxel grid data structure, completing a single dynamic update cycle of the panoramic 3D model of the working face.

Claims

1. A method for local updating of panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face, characterized in that, Includes the following steps: Global static initialization steps: Control the image acquisition device group in the fully mechanized mining face to synchronously acquire the initial frame image sequence, generate a global static reference model, and calibrate the initial external parameter matrix of each image acquisition device; Follow-up window definition steps: real-time acquisition of the real-time position coordinates of the coal mining machine uploaded by the coal mining machine position detection sensor, defining the follow-up window with the real-time position coordinates of the coal mining machine as the center in the global coordinate system, and activating the image acquisition device located in the follow-up window. Update trigger detection steps: Obtain the image features at the current time within the follow-up update window, calculate the spatial overlap rate by comparing the image features with the global static benchmark model, and generate an update trigger signal when the spatial overlap rate is lower than a preset threshold; Adaptive calibration steps for dynamic base: In response to the update trigger signal, acquire the support stroke detection sensor data of the hydraulic support located in the follow-up update window, calculate the cumulative displacement of the image acquisition device base by processing the support stroke detection sensor data, and use the cumulative displacement to correct the initial external parameter matrix to obtain the calibration external parameter matrix at the current moment; Local incremental reconstruction and fusion steps: The image data at the current moment is converted into a local point cloud using the calibration extrinsic parameter matrix, the local point cloud is registered to the global static reference model, and the registered local point cloud is fused into the global static reference model using a distance-weighted linear interpolation algorithm to complete the update of the panoramic 3D model.

2. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 1, characterized in that, In the global static initialization step, generating the global static benchmark model includes fusing point cloud data of the overlapping fields of view of adjacent image acquisition devices. The fusing process adopts a weighted calculation method based on observation quality. The weighted calculation method is as follows: for each observation point in the overlapping area, obtain the angle between the line-of-sight vector of each observation point and the main optical axis of the image acquisition device, as well as the Euclidean distance from the observation point to the optical center of the image acquisition device; calculate the fusion weight of the observation point by multiplying the reciprocal of the sum of the square of the angle and a preset constant with the reciprocal of the Euclidean distance.

3. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 1, characterized in that, In the update-triggered detection step, the calculation process of the spatial overlap rate includes: The image features at the current moment within the dynamic update window are back-projected to generate a three-dimensional feature point set; each feature point in the three-dimensional feature point set is traversed, and the corresponding point closest to the feature point is found in the surface point set of the global static reference model; The number of matching points is calculated by counting the number of points whose Euclidean distance between the feature point and the corresponding point is less than a preset judgment threshold. The spatial overlap rate is calculated by comparing the number of matching points with the total number of feature points in the three-dimensional feature point set.

4. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 1, characterized in that, In the adaptive calibration step of the moving base, the cumulative displacement is calculated as follows: Get the number of elements currently located within the dynamically updated window. The current stroke value of the support stroke detection sensor of the hydraulic support of the image acquisition device; Obtain the time when the global static initialization step is completed. The initial stroke value of the hydraulic support of the image acquisition device; The difference is calculated by subtracting the current travel value from the initial travel value, and the cumulative displacement of the image acquisition device base on the hydraulic support is calculated by multiplying the difference by a directional coefficient determined according to the installation direction of the pushing jack.

5. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 4, characterized in that, In the adaptive calibration step of the moving base, the process of correcting the initial external parameter matrix using the cumulative displacement includes: Construct a moving base compensation matrix, which is a homogeneous transformation matrix. The translation component of the homogeneous transformation matrix in the working face advancing direction is assigned the cumulative advancing amount, the other translation components are zero, and the rotation part is an identity matrix. The calibration extrinsic parameter matrix is ​​calculated by left-multiplying the dynamic base compensation matrix by the initial extrinsic parameter matrix.

6. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 5, characterized in that, In the local incremental reconstruction and fusion step, the process of generating a local point cloud and registering it using the calibration extrinsic parameter matrix includes: The original local point cloud of the image at the current time is generated using a depth estimation model; The coarse registration point cloud is calculated by multiplying the calibration extrinsic parameter matrix with the coordinate vector in the original local point cloud; The coarse registration point cloud uses the data from the support stroke detection sensor to eliminate the rigid displacement error caused by the hydraulic support movement.

7. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 6, characterized in that, After obtaining the coarsely registered point cloud, the registration process also includes a fine registration step: Select the overlapping region between the coarse registration point cloud and the global static reference model; The optimal rigid body transformation matrix is ​​calculated by using the iterative nearest point algorithm. The objective function of the iterative nearest point algorithm is set as follows: minimizing the sum of squared Euclidean distances between the points in the coarse registration point cloud after transformation by the transformation matrix to be solved and the corresponding nearest neighbor points in the global static benchmark model. The coarsely registered point cloud is transformed using the solved optimal rigid body transformation matrix to obtain the finely registered local point cloud.

8. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 1, characterized in that, In the local incremental reconstruction and fusion step, the distance-weighted linear interpolation algorithm calculates the fusion weight by determining the position of the spatial point relative to the center of the dynamically updated window. The rules for calculating the fusion weight are as follows: Define the core region radius and the edge radius, where the edge radius is larger than the core region radius; When the distance from a spatial point to the center of the dynamically updated window is less than or equal to the radius of the core region, the fusion weight is set to 1. When the distance from a spatial point to the center of the dynamically updated window is greater than the radius of the core region and less than the radius of the edge region, the fusion weight decreases linearly with the increase of distance. When the distance from a spatial point to the center of the dynamically updated window is greater than or equal to the edge radius, the fusion weight is set to 0.

9. The local update method for panoramic three-dimensional dynamic reconstruction of a coal mining machine-following working face according to claim 8, characterized in that, The process of integrating into the global static benchmark model includes: For each spatial location within the fusion region, the final spatial point coordinates are calculated by applying the fusion weights. The spatial point coordinates are calculated as follows: the first component is calculated by multiplying the coordinates of the registered local point cloud at the spatial location by the fusion weight; the second component is calculated by subtracting the fusion weight from 1 to calculate the difference, and then multiplying the difference by the original coordinates of the global static reference model at the spatial location. The calculation of the spatial point coordinates of the updated panoramic 3D model at the specified spatial location is achieved by adding the first component to the second component.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images