Cooperative perception and three-dimensional reconstruction method and system for unstructured rock targets

Abstract

The application discloses a cooperative perception and three-dimensional reconstruction method and system for unstructured rock targets, which comprises the following steps: pixel point cloud mapping is performed on a double-view depth map and a double-view RGB map of an unstructured rock target region to obtain a double-view rock target three-dimensional point cloud; an initial space variation matrix is constructed; an iterative closest point method and a gradual space registration strategy are adopted to optimize the initial space variation matrix to obtain an optimal space variation matrix; the double-view rock target three-dimensional point cloud is processed based on the optimal space variation matrix to obtain a unified rock target three-dimensional point cloud; surface normal estimation is performed on the unified rock target three-dimensional point cloud to obtain a surface normal vector; a relationship between the space position of the unified rock target three-dimensional point cloud and the surface normal vector is established based on a SPR surface reconstruction method to obtain a rock target three-dimensional model. The application improves the rock perception coverage, reconstruction integrity and adaptability by constructing a double-view cooperative observation and space unification mechanism.

Description

Technical Field

[0001] This invention relates to the field of intelligent mining technology, specifically to a collaborative perception and 3D reconstruction method and system for unstructured rock targets. Background Technology

[0002] With the rapid development of smart mines, unmanned mining, and industrial robot technologies, the demand for autonomous perception and intelligent operation in complex mining environments is becoming increasingly urgent. Rock targets are the primary objects in mine crushing, intelligent loading, and screening processes, and accurate acquisition of their spatial location, geometry, and surface structure is fundamental to achieving automated operations. Especially in automated rock breaking and intelligent grasping tasks, robots need to determine their operational position and movement path based on the three-dimensional morphology of the rock, placing high demands on the completeness and accuracy of the three-dimensional model.

[0003] Currently, 3D perception in mining scenarios primarily relies on RGB-D cameras, binocular vision, and laser scanning equipment. However, mining environments are typically unstructured: rocks are randomly piled up, varying greatly in size and shape, and easily come into contact, occlusion, and nesting with each other. In traditional single-view perception, sensors can only capture the visible area facing the lens; areas facing away or occluded cannot be directly observed, leading to problems such as local missing points and broken edges in point cloud data, making it difficult for the reconstructed model to reflect the true overall structure.

[0004] To expand coverage, existing research has gradually introduced multi-view collaborative sensing. However, due to differences in observation angle, surface coverage, and local details among different perspectives, achieving accurate alignment and effective fusion of point cloud data from different perspectives remains a core bottleneck. In addition, rock surfaces contain irregular curved surfaces and local abrupt changes, and the overlapping area of ​​data collected from different perspectives is limited, making it easy for registration errors, local structural shifts, and surface discontinuities to occur during the fusion process.

[0005] Meanwhile, the harsh conditions at the mine site, such as high dust levels, low light intensity, strong vibrations, and surface reflections, further introduce a large amount of noise and depth anomalies, seriously interfering with subsequent processing.

[0006] In summary, the existing technology mainly has the following problems: (1) The single-view perception range is limited, making it difficult to obtain the complete spatial structure of the rock, and the information of the obscured area is seriously lacking; (2) When fusing multi-view data, there is a lack of an effective spatial unified expression mechanism, making it difficult to make full use of complementary information between viewpoints; (3) Existing methods have limited adaptability to unstructured and irregular rocks, and it is difficult to balance model integrity, geometric accuracy and processing efficiency. (4) The three-dimensional reconstruction results are mostly general geometric models, lacking structured geometric parameter outputs for robot rock breaking, grasping and other operations. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a collaborative sensing and 3D reconstruction method and system for unstructured rock targets. By constructing a dual-view collaborative observation and spatial unification mechanism, it improves the rock sensing coverage, reconstruction integrity, and adaptability.

[0008] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0009] According to a first aspect of this application, a collaborative sensing and 3D reconstruction method for unstructured rock targets is provided, comprising: Pixel point cloud mapping is performed on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view three-dimensional point cloud of the rock target. An initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud is constructed. The initial spatial variation matrix is ​​optimized by the iterative nearest point method and the progressive spatial registration strategy to obtain the optimal spatial variation matrix. Based on the optimal spatial variation matrix, the dual-view rock target 3D point cloud is processed to obtain a spatially aligned unified rock target 3D point cloud. The surface normal vector is estimated by performing surface normal estimation on the unified rock target 3D point cloud. Based on the SPR surface reconstruction method, the relationship between the spatial position of the unified rock target 3D point cloud and the surface normal vector is established to obtain the rock target 3D model.

[0010] Furthermore, the step of mapping pixel point clouds from the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain a dual-view three-dimensional point cloud of the rock target includes: Rock target detection and segmentation are performed on dual-view RGB images to obtain dual-view target region masks; Dual-view depth is obtained based on dual-view depth map. Based on dual-view depth and camera intrinsic parameter matrix, pixel point cloud mapping is performed on the dual-view target area mask to obtain dual-view rock target 3D point cloud.

[0011] Furthermore, before processing the 3D point cloud of the dual-view rock target based on the optimal spatial variation matrix, the process further includes: A method combining statistical outlier removal and radius filtering is used to remove dust interference points, edge anomalies, and depth measurement error points from the three-dimensional point cloud of the dual-view rock target.

[0012] Furthermore, the dual-view rock target 3D point cloud includes a first-view rock target 3D point cloud and a second-view rock target 3D point cloud. The initial spatial transformation matrix for registering the dual-view rock target 3D point cloud includes: Based on the dual-view calibration results, an initial spatial transformation matrix is ​​established to map the three-dimensional point cloud of the rock target from the first view to the three-dimensional point cloud of the rock target from the second view.

[0013] Furthermore, the initial spatial transformation matrix is ​​optimized using an iterative nearest-point method, including: By repeatedly establishing the point correspondence between the 3D point cloud of the dual-view rock target and the reference point cloud, and minimizing the Euclidean distance error between the points, the spatial transformation matrix is ​​iteratively optimized. The objective function is expressed as: in, This represents spatial points in the point cloud to be registered. The point cloud to be registered consists of a first-view 3D point cloud of a rock target and a second-view 3D point cloud of a rock target. This represents the corresponding point in the reference point cloud. This represents the optimized spatial transformation matrix. This is the initial spatial transformation matrix.

[0014] Furthermore, a progressive spatial registration strategy is employed to optimize the initial spatial transformation matrix, including: In the initial stage of registration, a preset search radius for corresponding points is used as the search radius to search for corresponding points; As the number of iterations increases, the search radius is gradually reduced, causing the corresponding points to gradually concentrate in the true matching region. The search radius update method is expressed as follows: in: For the first The search radius at the next iteration The search radius of the previous iteration. Let be the search radius decay coefficient, satisfying ; In each iteration, the spatial transformation matrix is ​​recalculated based on the new correspondence between points, and the current registration error is calculated. When the error change is less than a preset threshold for two consecutive iterations, the algorithm is considered to have reached the convergence condition, and the final spatial transformation matrix is ​​output.

[0015] Furthermore, the process of estimating the surface normal vector of the unified rock target's 3D point cloud to obtain the surface normal vector, and establishing the relationship between the spatial position of the unified rock target's 3D point cloud and the surface normal vector based on the SPR surface reconstruction method to obtain the 3D model of the rock target, includes: By using a statistical outlier removal method, outliers are identified and removed from the 3D point cloud of a uniform rock target based on the distribution characteristics of the point cloud neighborhood. For any point in the unified rock target 3D point cloud after removing outliers, a local neighborhood point set is established for the arbitrary point, the neighborhood spatial distribution relationship is analyzed, and the surface normal information corresponding to each point in the unified rock target 3D point cloud is obtained. All the surface normal information is then set into a surface normal vector. The surface reconstruction method based on SPR establishes the relationship between the spatial position of the unified 3D point cloud of the rock target and the surface normal vector as a continuous implicit surface, expressed as: in: This represents a continuous implicit curve to be solved. This represents a vector field constructed from surface normals. Represents the Laplace operator. Represents divergence operations; Solving the continuous implicit surface yields the continuous surface of the rock target. A triangular mesh model is extracted from this continuous surface, and a three-dimensional model of the rock target is reconstructed. The expression is: in: This represents the set of vertices of a 3D model of a rock target. This represents the edge connection relationships of the 3D model of the rock target. This represents a set of triangular facets representing a three-dimensional model of a rock target.

[0016] Furthermore, it also includes geometric analysis of the three-dimensional model of the rock target, specifically: Obtain the spatial centroid of the rock target based on its 3D model: in: Let the geometric centroid of the rock target be... For the first One vertex, This represents the total number of vertices in the model. Establish the covariance matrix based on the spatial centroid of the rock target and the vertices of the 3D model of the rock target: Eigenvalue decomposition is performed on the covariance matrix to obtain three corresponding eigenvectors. The spatial attitude of the rock target is estimated by the three eigenvectors. The eigenvector corresponding to the largest eigenvalue is used as the principal direction of the rock target, and the other two eigenvectors together constitute the local coordinate system of the target. Based on the spatial attitude estimation of the rock target, the maximum and minimum coordinate values ​​of the three-dimensional model of the rock target in the three principal axis directions are obtained, and the spatial dimension parameters such as the length, width and height of the rock target are obtained. The minimum bounding box of the target is determined to estimate the size and volume of the rock target. Obtain the surface normal vectors corresponding to each triangular facet in the 3D model of the rock target based on the triangular mesh model.

[0017] According to a second aspect of this application, a collaborative sensing and 3D reconstruction system for unstructured rock targets is provided, comprising: The first 3D point cloud acquisition module is used to perform pixel point cloud mapping on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view 3D point cloud of the rock target. The second 3D point cloud acquisition module is used to construct an initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud. The initial spatial variation matrix is ​​optimized by using an iterative nearest point method and a progressive spatial registration strategy to obtain an optimal spatial variation matrix. Based on the optimal spatial variation matrix, the dual-view rock target 3D point cloud is processed to obtain a spatially aligned unified rock target 3D point cloud. The 3D model acquisition module for rock targets is used to estimate the surface normal of the unified 3D point cloud of the rock target to obtain the surface normal vector. Based on the SPR surface reconstruction method, the relationship between the spatial position of the unified 3D point cloud of the rock target and the surface normal vector is established to obtain the 3D model of the rock target.

[0018] According to a third aspect of this application, an electronic device is provided, comprising: One or more processors; A memory for storing executable instructions of the processor, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0019] The beneficial effects of this application are as follows: (1) The collaborative perception and 3D reconstruction method and system for unstructured rock targets provided in this application solves the problem of missing 3D information of rock targets caused by occlusion and stacking in unstructured mining environments through an integrated mechanism of dual-view collaborative observation, spatial unification, fusion reconstruction and geometric analysis. Compared with traditional single-view methods, this application significantly improves the integrity and surface continuity of the 3D model, and achieves a qualitative transformation from messy point clouds to executable spatial information by extracting structured geometric parameters for robots, thereby improving the success rate of intelligent rock breaking and grasping operations.

[0020] (2) The collaborative perception and 3D reconstruction method and system for unstructured rock targets provided in this application adopts a dual-view spatial complementarity mechanism, which expands the surface coverage from a physical level and reduces information blind spots from the source. In the data fusion stage, the consistency of multi-source data is ensured by establishing unified spatial coordinate constraints instead of relying on a single global optimization search. Finally, the reconstruction model is deconstructed into position, principal direction and normal vector distribution, which directly eliminates the need for secondary calculation of the original grid data by the robot actuator during path planning, thereby significantly reducing decision delay and improving positioning accuracy.

[0021] (3) The collaborative perception and three-dimensional reconstruction method and system for unstructured rock targets provided in this application effectively reduce the limitation of single-view field of view and improve the integrity of spatial data acquisition through the dual-view collaborative acquisition mechanism; and ensure the accuracy of target extraction under low dust illumination through 2D-3D joint mapping; in addition, ensure the stability of multi-view data fusion through the unified expression mechanism constructed by the spatial alignment module.

[0022] (4) The collaborative perception and three-dimensional reconstruction method and system for unstructured rock targets provided in this application integrates the reconstruction module to restore the missing structure using complementary information and improve the surface quality of the model; the geometric analysis module extracts the main direction, bounding box and normal, providing direct parameters for rock breaking and grasping.

[0023] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and are intended to explain the invention, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of a collaborative sensing and 3D reconstruction method for unstructured rock targets according to the present invention; Figure 2 This is a schematic diagram of a collaborative sensing and 3D reconstruction framework for unstructured rock targets according to the present invention. Figure 3 This is a schematic diagram of an RGB image in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of a depth map in a specific embodiment of the present invention; Figure 5 This is a schematic diagram of a specific embodiment of the present invention before noise reduction; Figure 6 This is a schematic diagram of the denoised version of a specific embodiment of the present invention; Figure 7This is a schematic diagram of the segmentation of a first instance according to a specific embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the segmentation of a second instance according to a specific embodiment of the present invention; Figure 9 This is a schematic diagram of point cloud registration according to a specific embodiment of the present invention; Figure 10 This is a schematic diagram of a three-dimensional model of a rock target in a specific embodiment of the present invention; Figure 11 This is a flowchart of a collaborative sensing and 3D reconstruction system for unstructured rock targets according to the present invention; Figure 12 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0025] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0026] It should be understood that the terms "comprising" and other similar expressions in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, or apparatus that includes a series of steps or units and is not limited to the listed steps or units. Furthermore, "first" and "second" are used to distinguish different objects and are not intended to describe a specific order.

[0027] According to the first aspect of this application, Figure 1 and Figure 2 As shown, this embodiment provides a collaborative sensing and 3D reconstruction method for unstructured rock targets, including: Step S1: Perform pixel point cloud mapping on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view three-dimensional point cloud of the rock target.

[0028] In this embodiment, two industrial 3D cameras are used to form a collaborative observation structure with a fixed spatial angle, creating spatial complementarity between the viewpoints and forming a dual-view industrial 3D camera system. This dual-view industrial 3D camera system includes a first-view industrial 3D camera and a second-view industrial 3D camera. The first-view industrial 3D camera is one of the left and right industrial 3D cameras, and the second-view industrial 3D camera is the other of the left and right industrial 3D cameras. The dual-view industrial 3D camera system acquires 2D image information and 3D spatial information of unstructured rock targets under different observation directions and establishes initial spatial relationships between different viewpoints, providing basic data for subsequent rock target perception, spatial alignment, and 3D fusion reconstruction. A unified triggering mechanism ensures temporal consistency, thereby expanding surface coverage at the source and reducing information loss caused by random accumulation and occlusion. Figure 3 The image shown is a schematic diagram of an RGB image in a specific embodiment of the present invention. Figure 4 The image shown is a schematic diagram of a depth map in a specific embodiment of the present invention.

[0029] Specifically, the dual-view industrial 3D camera can simultaneously acquire dual-view depth maps and dual-view RGB maps of the target area. The RGB image is used to describe the color, texture, and edge information of the rock target, while the depth map is used to recover the spatial structure information of the target surface.

[0030] In some embodiments of this example, the step of mapping pixel point clouds of the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain a dual-view three-dimensional point cloud of the rock target includes: Rock target detection and segmentation are performed on dual-view RGB images to obtain dual-view target region masks; Dual-view depth is obtained based on dual-view depth map. Based on dual-view depth and camera intrinsic parameter matrix, pixel point cloud mapping is performed on the dual-view target area mask to obtain dual-view rock target 3D point cloud.

[0031] During actual data acquisition, the fields of view of the two cameras partially overlapped. This overlapping area contained common spatial information about the same rock surface, which could be used for subsequent spatial consistency constraints and point cloud fusion. The non-overlapping areas, on the other hand, supplemented the side and edge regions that could not be captured by a single viewpoint, improving the target surface coverage. Simultaneously, to ensure temporal consistency of the dual-view data, the two industrial 3D cameras used a unified triggering method for synchronous acquisition, ensuring that data from different perspectives had the same time reference and reducing the impact of dynamic environmental changes on spatial information matching.

[0032] Through the above dual-view data acquisition process, this embodiment can obtain multi-source perception data containing rock target texture information, depth information, and spatial coordinate relationships, providing a reliable foundation for subsequent rock target region extraction, dual-view spatial alignment, and complete 3D model reconstruction.

[0033] The collaborative perception and 3D reconstruction method for unstructured rock targets provided in this embodiment is mainly used in mining industrial environments with low illumination, high dust, and complex obstructions. Therefore, it is necessary to preprocess, identify, and spatially map the image data acquired from dual perspectives to achieve accurate extraction of rock target areas and generation of 3D point clouds, providing high-quality input data for subsequent dual-view spatial alignment and 3D fusion reconstruction.

[0034] Mining operations, such as screening, crushing stations, and ore pass entrances, commonly suffer from insufficient lighting, pervasive dust, complex background structures, and overlapping and occlusion of rock targets. This results in raw images often exhibiting low contrast, blurred edges, weakened texture information, and severe noise interference, thus affecting target recognition accuracy and 3D reconstruction quality. Therefore, this embodiment first performs background interference suppression and target region correction on the RGB images acquired from dual perspectives.

[0035] Specifically, a brightness correction method is used to improve the overall brightness distribution of images under low-light conditions, while a local contrast enhancement method is employed to improve the discernibility of details and edge features on the rock surface. To address image degradation caused by dust particle scattering, a combination of filtering and image restoration is used to suppress random noise interference and improve image clarity and target boundary integrity. For example... Figure 5 The diagram shown is a schematic representation of a specific embodiment of the present invention before noise reduction; as shown... Figure 6 The diagram shown is a denoised schematic of a specific embodiment of the present invention.

[0036] Let the RGB image be After enhancement processing, the following is obtained: in: For the enhanced image, This represents the function for enhancing bottom illumination and suppressing dust.

[0037] Specifically, the background illumination enhancement processing function is usually a retina theory method, an adaptive histogram equalization method, or a deep learning method; the dust suppression processing function is usually a dark channel prior method or a color attenuation prior method.

[0038] The preprocessed image is used for rock target instance segmentation. In this embodiment, a deep learning-based target detection and segmentation method is used to identify rock targets and output the corresponding target region mask information. Figure 7The diagram shown is a schematic representation of the first instance segmentation in a specific embodiment of the present invention; as shown... Figure 8 The diagram shown is a schematic representation of the second instance segmentation in a specific embodiment of the present invention.

[0039] Specifically, for a rock target in the input image, a target mask is obtained after processing: in: Indicates the target area of ​​the rock. Indicates the background area.

[0040] Target masking enables effective separation of the rock target from the sieve structure, supporting components, and background environment, thereby obtaining pixel-level regional information of the rock target. Combined with depth map data, the target region is mapped to a three-dimensional spatial coordinate system. For each pixel... and their corresponding depth values According to the camera intrinsic parameter matrix .

[0041] Before implementing the collaborative perception and 3D reconstruction method for unstructured rock targets provided in this embodiment, camera calibration is performed on the dual-view industrial 3D camera, including calibration of the internal parameters of the single camera and calibration of the external parameters between the two cameras.

[0042] The imaging model of a single camera is represented as: in: Represents the pixel coordinates of the image. This represents the three-dimensional coordinates of the target point in the camera coordinate system. Represents the camera intrinsic parameter matrix. This represents the depth scale factor.

[0043] Focal length, principal point coordinates, and distortion parameters are obtained through camera intrinsic parameter calibration, thereby establishing a mapping relationship between 2D image coordinates and 3D spatial coordinates. After calibration, pixels are converted into spatial point clouds based on the acquired depth map. in: Indicates pixel position The corresponding depth value, This represents the generated three-dimensional spatial points.

[0044] Three-dimensional spatial coordinates can be obtained: After the above transformation, a three-dimensional point cloud of the rock target from two perspectives can be obtained: in: and These represent the three-dimensional point cloud sets of rock targets from the first-view perspective and the two-dimensional point cloud sets of rock targets from the second-view perspective, respectively.

[0045] To improve the point cloud quality in complex environments, this embodiment filters the point cloud based on a target mask, retaining only valid spatial points belonging to the rock target area, and uses a combination of statistical outlier removal (SOR) and radius filtering to remove dust interference points, edge anomalies, and depth measurement error points.

[0046] Let the set of three-dimensional point clouds of rock targets from a certain perspective be . The filtered 3D point cloud set of rock targets is represented as ,in This indicates an abnormal noise point.

[0047] After the above target recognition, spatial mapping and point cloud purification processes, the three-dimensional point clouds of the rock target from the first perspective and the three-dimensional point clouds of the rock target from the second perspective are obtained respectively, providing a high-quality input basis for subsequent dual-view spatial alignment and three-dimensional fusion reconstruction.

[0048] Thus, through the above processing mechanism, this embodiment can achieve stable perception of rock targets in low-light, high-dust, and complex occlusion environments, effectively improving the accuracy of target boundary extraction and the quality of point cloud data, thereby enhancing the stability and integrity of the subsequent 3D reconstruction process.

[0049] This solution addresses the limitations and occlusion issues of single-viewpoint fields of view by using two 3D cameras with a fixed spatial relationship to form a collaborative observation structure, leveraging the complementary spatial relationship between viewpoints to expand target surface coverage, and combining synchronous triggering to ensure data temporal consistency.

[0050] Furthermore, under low illumination and high dust conditions, a rock target region extraction method combining image preprocessing enhancement, deep learning target detection and segmentation with depth map coordinate mapping, and a precise point cloud generation strategy based on target masks are proposed.

[0051] Step S2: Construct the initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud. Optimize the initial spatial variation matrix using the iterative nearest point method and progressive spatial registration strategy to obtain the optimal spatial variation matrix. Based on the optimal spatial variation matrix, process the dual-view rock target 3D point cloud to obtain a spatially aligned unified rock target 3D point cloud. In this way, establish the spatial correspondence between rock target point clouds under different observation perspectives, achieve accurate registration of dual-view point cloud data in a unified coordinate system, and provide a consistent spatial reference for subsequent 3D data fusion and model reconstruction.

[0052] In some embodiments of this example, before processing the 3D point cloud of the dual-view rock target based on the optimal spatial transformation matrix, the method further includes: A method combining statistical outlier removal and radius filtering is used to remove dust interference points, edge anomalies, and depth measurement error points from the three-dimensional point cloud of the dual-view rock target.

[0053] In some embodiments of this example, the three-dimensional point cloud of the rock target in the first view is one of the three-dimensional point clouds of the rock target in the left and right view, and the three-dimensional point cloud of the rock target in the second view is another of the three-dimensional point clouds of the rock target in the left and right view.

[0054] like Figure 2 As shown, the dual-view rock target 3D point cloud includes a first-view rock target 3D point cloud and a second-view rock target 3D point cloud. The initial spatial transformation matrix for registering the dual-view rock target 3D point cloud, to achieve initial registration, includes: Based on the dual-view calibration results, an initial spatial transformation matrix is ​​established to map the three-dimensional point cloud of the rock target from the first view to the three-dimensional point cloud of the rock target from the second view.

[0055] Specifically, the first-person view target point cloud is The second-person view target point cloud is Based on the dual-view calibration results, an initial spatial transformation matrix is ​​established between the first-view and second-view 3D point clouds of the rock target: in: Represents the rotation matrix. Represents the translation vector. This indicates the spatial transformation relationship between the second-view coordinate system and the first-view coordinate system.

[0056] Based on the initial transformation matrix, map the second-view target point cloud to the first-view target point cloud reference coordinate system: Complete the initial coarse registration of the point cloud from both perspectives.

[0057] In some implementations of this embodiment, the initial spatial transformation matrix is ​​optimized using an iterative nearest-point method, including: By repeatedly establishing the point correspondence between the 3D point cloud of the dual-view rock target and the reference point cloud, and minimizing the Euclidean distance error between the points, the spatial transformation matrix is ​​iteratively optimized. The objective function is expressed as: in, This represents spatial points in the point cloud to be registered. The point cloud to be registered consists of a first-view 3D point cloud of a rock target and a second-view 3D point cloud of a rock target. This represents the corresponding point in the reference point cloud, which is obtained by calibrating the unstructured rock target area. This represents the optimized spatial transformation matrix. This is the initial spatial transformation matrix.

[0058] In some implementations of this embodiment, such as Figure 2 As shown, considering the characteristics of unstructured rock targets such as local occlusion, surface discontinuities, and uneven point cloud density distribution, this embodiment adopts a progressive spatial registration strategy to improve the convergence stability of the ICP algorithm. In the initial stage of registration, a larger search radius for corresponding points is used to increase the search range; as the number of iterations increases, the search radius is gradually reduced, causing the corresponding points to gradually concentrate in the true matching region, thereby improving the final spatial registration accuracy. Specifically, the progressive spatial registration strategy optimizes the initial spatial transformation matrix to achieve fine registration, including: In the initial stage of registration, a preset search radius for corresponding points is used as the search radius to search for corresponding points; As the number of iterations increases, the search radius is gradually reduced, causing the corresponding points to gradually concentrate in the true matching region. The search radius update method is expressed as follows: in: For the first The search radius at the next iteration The search radius of the previous iteration. Let be the search radius decay coefficient, satisfying ; In each iteration, the spatial transformation matrix is ​​recalculated based on the new correspondence between points, and the current registration error is calculated. When the error change is less than a preset threshold for two consecutive iterations, the algorithm is considered to have reached convergence, and the final spatial transformation matrix is ​​output. After fine registration, the optimized spatial transformation matrix is ​​used to fuse the 3D point clouds of the first-view and second-view rock targets, achieving a spatial representation of the dual-view point clouds in a unified coordinate system. in: To achieve a unified 3D point cloud of the rock target after spatial alignment.

[0059] like Figure 9The diagram shown illustrates point cloud registration in a specific embodiment of the present invention. Thus, through the aforementioned dual-view spatial alignment process, this embodiment fully utilizes the complementary relationship between the observation information from both perspectives, constructs an initial unified coordinate system using calibrated extrinsic parameters, and combines a progressive constraint optimization strategy to achieve accurate registration of rock target point cloud data under different observation directions. This effectively reduces spatial deviations caused by differences in observation angles, local occlusion, and depth measurement errors, corrects dynamic drift, improves the spatial consistency of point cloud data and target surface coverage, and provides a unified and accurate spatial data foundation for subsequent 3D data fusion and reconstruction.

[0060] Step S3: Estimate the surface normal of the unified rock target 3D point cloud to obtain the surface normal vector. Based on the SPR surface reconstruction method, establish the relationship between the spatial position of the unified rock target 3D point cloud and the surface normal vector to obtain the rock target 3D model.

[0061] Step S3 in this embodiment is used to perform fusion processing and 3D surface reconstruction on the unified 3D point cloud of the rock target after spatial alignment. It fuses the local spatial information obtained from different observation directions into a continuous and complete 3D model of the rock target, realizing the transformation of unstructured rock targets from discrete point cloud data to digital 3D entities, and providing a reliable spatial data foundation for subsequent rock geometric feature extraction, target posture analysis and robot autonomous operation.

[0062] In complex mining environments, rock targets are often characterized by random accumulation, irregular shape, and mutual occlusion. Point clouds obtained from a single viewpoint can only describe the visible area of ​​the target, which can easily lead to the loss of spatial information on the sides, edges, and occluded areas of the rock target. This can result in problems such as local voids, surface discontinuities, and incomplete overall structure in the reconstruction model.

[0063] Although dual-view spatial alignment can transform rock target point clouds acquired from different perspectives into a unified coordinate system and expand the target surface coverage, the fused point cloud data still belongs to discrete spatial sampling points and cannot directly represent the continuous curved surface structure of the rock target. Therefore, this embodiment further constructs a three-dimensional model of the rock target, optimizes the fused target point cloud, and restores the complete three-dimensional morphology of the rock target through surface reconstruction methods.

[0064] like Figure 2As shown, in some embodiments of this example, due to factors such as dust interference, depth measurement errors, and local abnormal reflections at the mine site, the unified rock target 3D point cloud may contain a small number of outliers and noise points. Therefore, this step first preprocesses the unified rock target 3D point cloud. Using the Statistical Outlier Removal (SOR) method, outliers are identified based on the neighborhood distribution characteristics of the point cloud and removed from the unified rock target 3D point cloud to improve the point cloud data quality. Then, surface normals are estimated on the unified rock target 3D point cloud after removing outliers to obtain surface normal vectors. Based on the SPR surface reconstruction method, the relationship between the spatial location of the unified rock target 3D point cloud and the surface normal vectors is established to obtain the 3D model of the rock target, including: Receive the unified 3D point cloud of the rock target after spatial alignment. , The spatial points of the rock target are represented by a statistical outlier removal method. Based on the distribution characteristics of the point cloud neighborhood, outliers are identified and removed from the 3D point cloud of the unified rock target. For any point in the unified rock target 3D point cloud after removing outliers, surface normal estimation is performed. Specifically, by establishing a set of local neighborhood points for any point, analyzing the spatial distribution relationship of the neighborhood, obtaining the surface normal information corresponding to each point in the unified rock target 3D point cloud, and collecting all the surface normal information into a surface normal vector, the surface normal vector can describe the local surface direction of the rock target, providing a geometric basis for subsequent continuous surface restoration. After removing outliers from the unified 3D point cloud and surface normal vectors of the rock target, the surface reconstruction method based on SPR (Screened Poisson Reconstruction) establishes the relationship between the spatial position of the unified rock target's 3D point cloud and the surface normal vectors as a continuous implicit surface, expressed as: in: This represents a continuous implicit curve to be solved. This represents a vector field constructed from surface normals. Represents the Laplace operator. Represents divergence operations; Solving the continuous implicit surface yields the continuous surface of the rock target. A triangular mesh model is extracted from this continuous surface, and a three-dimensional model of the rock target is reconstructed. The expression is: in: This represents the set of vertices of a 3D model of a rock target. This represents the edge connection relationships of the 3D model of the rock target. This represents a set of triangular facets representing a three-dimensional model of a rock target.

[0065] This three-dimensional model of the rock target not only contains the overall spatial morphology information of the rock target, but also preserves the topological structure and geometric relationship of the target surface, realizing the digital expression of the true morphology of unstructured rock targets.

[0066] like Figure 10 As shown, this is a schematic diagram of a three-dimensional rock target model in a specific embodiment of the present invention. In this way, the aligned point clouds from different perspectives are regarded as complementary expressions of information about the same target. The occlusion and missing areas are restored through spatial complementarity, and a continuous and complete three-dimensional model is generated by combining surface reconstruction.

[0067] In some embodiments of this example, in order to improve the model quality, the generated three-dimensional rock target model is also optimized, including local mesh smoothing, abnormal patch correction and hole area repair, etc., to improve the continuity of the model surface and structural integrity.

[0068] After the above processing, a complete 3D model of the rock target is finally obtained. Compared with the single-view reconstruction result, this embodiment fully utilizes the complementary relationship between information from different observation directions through dual-view data fusion, effectively recovering the rock sides and occluded areas that cannot be obtained from a single view, thus improving the integrity and spatial representation accuracy of the target 3D model.

[0069] Finally, the optimized 3D model of the rock target is output, which includes the fused point cloud data of the rock target, the 3D surface model of the rock target, the triangular mesh model of the rock target, and the spatial topology information of the target.

[0070] The above data serves as input to the subsequent target geometric feature extraction module, used to further calculate parameters such as the position, size, attitude, principal direction, and surface orientation of the rock target.

[0071] Through the 3D fusion reconstruction process of rock targets, this embodiment realizes the conversion from dual-view spatial perception data to a complete 3D model, reduces the impact of occlusion, noise and data loss in complex mining environments on the reconstruction results, improves the integrity, accuracy and reliability of 3D reconstruction of unstructured rock targets, and provides a high-quality spatial perception foundation for robots to automatically break rocks, intelligently grasp and operate autonomously.

[0072] The target geometric feature extraction module is used to perform geometric feature analysis on the 3D model of the rock target generated by the 3D data fusion and reconstruction module, and extract key parameters such as the spatial position, geometric centroid, principal direction, attitude and surface normal of the rock target, so as to provide reliable spatial information support for the robot's automatic rock breaking, intelligent grasping, path planning and autonomous operation decision-making.

[0073] like Figure 2 As shown, after 3D data fusion and reconstruction, a 3D model of a rock target with a continuous surface structure is obtained. Based on this, the process also includes geometric analysis of the 3D rock target model, calculating the geometric properties of the rock target, and realizing the conversion of the target from a 3D model to engineering application parameters. Specifically: Calculate the spatial centroid of the rock target based on its 3D model. Assume the 3D model of the rock target contains... There are vertices, and their vertex set is represented as . Then the geometric centroid of the rock target can be represented as: in: Let the geometric centroid of the rock target be... For the first One vertex, This represents the total number of vertices in the model.

[0074] The geometric centroid reflects the overall spatial position of the rock target, providing a reference for robot target localization and grasping planning. Subsequently, to obtain the spatial pose of the rock target, a covariance matrix is ​​established based on the vertices of the rock target's 3D model: The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three corresponding eigenvectors. The eigenvector corresponding to the largest eigenvalue is used as the principal direction of the rock target, and the other two eigenvectors together form the local coordinate system of the target, thus realizing the spatial attitude estimation of the rock target.

[0075] After obtaining the target's attitude, the spatial dimension parameters such as the length, width, and height of the rock target are calculated based on the maximum and minimum coordinate values ​​of the three-dimensional model of the rock target in the three principal axis directions. Furthermore, the target's minimum bounding box (OBB) is determined to achieve the volume estimation of the rock target's size.

[0076] To describe the surface orientation information of the rock target, the surface normal vector corresponding to each triangular facet is calculated based on the triangular mesh model. Let the three vertices of the triangular facet be... , and The formula for calculating the surface normal vector is: in: Let be the unit normal vector of the triangular facet. This represents the vector cross product operation. This represents the magnitude of the vector.

[0077] By statistically analyzing the distribution of the target surface normal vectors, the main force direction and surface spatial orientation of the rock target can be obtained, providing a geometric basis for planning the strike direction during subsequent automatic rock breaking.

[0078] Finally, the system outputs geometric feature parameters of the rock target, such as its spatial position, orientation, three-dimensional dimensions, principal orientation, and surface normal, providing accurate and reliable spatial information support for the robot's automatic rock breaking, intelligent grasping, path planning, and autonomous operation decision-making.

[0079] Through the aforementioned target geometric feature extraction process, this embodiment can further convert the reconstructed 3D model into geometric parameter information that the robot can directly utilize. This enables data connection from 3D perception of unstructured rock targets to intelligent operational decision-making, improving the robot's autonomous perception capability and operational accuracy in complex mining environments. Furthermore, based on the extracted centroid, principal orientation, minimum bounding box size, and surface normal distribution from the reconstructed model, the 3D mesh data is converted into structured parameters that can be directly used by the robot for positioning, grasping, and rock breaking. The dual-view acquisition, target perception, spatial unification, complementary fusion, and geometric analysis are then integrated at the system level, forming a complete technical chain from raw visual input to robot-executable spatial commands.

[0080] According to the second aspect of this application, such as Figure 11 As shown in the figure, this embodiment provides a collaborative sensing and 3D reconstruction system for unstructured rock targets, including: The first 3D point cloud acquisition module is used to perform pixel point cloud mapping on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view 3D point cloud of the rock target. The second 3D point cloud acquisition module is used to construct an initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud. The initial spatial variation matrix is ​​optimized by using an iterative nearest point method and a progressive spatial registration strategy to obtain an optimal spatial variation matrix. Based on the optimal spatial variation matrix, the dual-view rock target 3D point cloud is processed to obtain a spatially aligned unified rock target 3D point cloud. The 3D model acquisition module for rock targets is used to estimate the surface normal of the unified 3D point cloud of the rock target to obtain the surface normal vector. Based on the SPR surface reconstruction method, the relationship between the spatial position of the unified 3D point cloud of the rock target and the surface normal vector is established to obtain the 3D model of the rock target.

[0081] Specifically, this embodiment corresponds one-to-one with the above method embodiments. The functions of each module have been described in detail in the corresponding method embodiments, so they will not be repeated here.

[0082] According to a third aspect of this application, this embodiment provides an electronic device, specifically, as follows: Figure 12 As shown, Figure 12 The diagram shown is a schematic diagram of an electronic device in one embodiment of this invention. It is a structural block diagram of the electronic device provided in this embodiment of the invention. The electronic device provided in this embodiment of the invention includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps as described in the above method embodiment.

[0083] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0084] The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the schematic diagram is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0085] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0086] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0087] If the modules integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0088] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0089] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the steps of the method as described in the above embodiments.

[0090] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A collaborative sensing and 3D reconstruction method for unstructured rock targets, characterized in that, include: Pixel point cloud mapping is performed on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view three-dimensional point cloud of the rock target. An initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud is constructed. The initial spatial variation matrix is ​​optimized by the iterative nearest point method and the progressive spatial registration strategy to obtain the optimal spatial variation matrix. Based on the optimal spatial variation matrix, the dual-view rock target 3D point cloud is processed to obtain a spatially aligned unified rock target 3D point cloud. The surface normal vector is estimated by performing surface normal estimation on the unified rock target 3D point cloud. Based on the SPR surface reconstruction method, the relationship between the spatial position of the unified rock target 3D point cloud and the surface normal vector is established to obtain the rock target 3D model.

2. The method according to claim 1, characterized in that, The process of mapping pixels into point clouds from dual-view depth maps and dual-view RGB maps of unstructured rock target areas to obtain dual-view 3D point clouds of rock targets includes: Rock target detection and segmentation are performed on dual-view RGB images to obtain dual-view target region masks; Dual-view depth is obtained based on dual-view depth map. Based on dual-view depth and camera intrinsic parameter matrix, pixel point cloud mapping is performed on the dual-view target area mask to obtain dual-view rock target 3D point cloud.

3. The method according to claim 1, characterized in that, Before processing the 3D point cloud of the dual-view rock target based on the optimal spatial transformation matrix, the process further includes: A method combining statistical outlier removal and radius filtering is used to remove dust interference points, edge anomalies, and depth measurement error points from the three-dimensional point cloud of the dual-view rock target.

4. The method according to claim 1, characterized in that, The dual-view rock target 3D point cloud includes a first-view rock target 3D point cloud and a second-view rock target 3D point cloud. The initial spatial transformation matrix for registering the dual-view rock target 3D point cloud includes: Based on the dual-view calibration results, an initial spatial transformation matrix is ​​established to map the three-dimensional point cloud of the rock target from the first view to the three-dimensional point cloud of the rock target from the second view.

5. The method according to claim 1, characterized in that, The initial spatial transformation matrix is ​​optimized using the iterative nearest-point method, including: By repeatedly establishing the point correspondence between the 3D point cloud of the dual-view rock target and the reference point cloud, and minimizing the Euclidean distance error between the points, the spatial transformation matrix is ​​iteratively optimized. The objective function is expressed as: in, This represents spatial points in the point cloud to be registered. The point cloud to be registered consists of a first-view 3D point cloud of a rock target and a second-view 3D point cloud of a rock target. This represents the corresponding point in the reference point cloud. This represents the optimized spatial transformation matrix. This is the initial spatial transformation matrix.

6. The method according to claim 5, characterized in that, The initial spatial transformation matrix is ​​optimized using a progressive spatial registration strategy, including: In the initial stage of registration, a preset search radius for corresponding points is used as the search radius to search for corresponding points; As the number of iterations increases, the search radius is gradually reduced, causing the corresponding points to gradually concentrate in the true matching region. The search radius update method is expressed as follows: in: For the first The search radius at the next iteration The search radius of the previous iteration. Let be the search radius decay coefficient, satisfying ; In each iteration, the spatial transformation matrix is ​​recalculated based on the new correspondence between points, and the current registration error is calculated. When the error change is less than a preset threshold for two consecutive iterations, the algorithm is considered to have reached the convergence condition, and the final spatial transformation matrix is ​​output.

7. The method according to claim 1, characterized in that, The process involves estimating the surface normal vector of the unified rock target's 3D point cloud, obtaining the surface normal vector, and establishing the relationship between the spatial position of the unified rock target's 3D point cloud and the surface normal vector using the SPR surface reconstruction method to obtain the 3D model of the rock target. This includes: By using a statistical outlier removal method, outliers are identified and removed from the 3D point cloud of a uniform rock target based on the distribution characteristics of the point cloud neighborhood. For any point in the unified rock target 3D point cloud after removing outliers, a local neighborhood point set is established for the arbitrary point, the neighborhood spatial distribution relationship is analyzed, and the surface normal information corresponding to each point in the unified rock target 3D point cloud is obtained. All the surface normal information is then set into a surface normal vector. The surface reconstruction method based on SPR establishes the relationship between the spatial position of the unified 3D point cloud of the rock target and the surface normal vector as a continuous implicit surface, expressed as: in: This represents a continuous implicit curve to be solved. This represents a vector field constructed from surface normals. Represents the Laplace operator. Represents divergence operations; Solving the continuous implicit surface yields the continuous surface of the rock target. A triangular mesh model is extracted from this continuous surface, and a three-dimensional model of the rock target is reconstructed. The expression is: in: This represents the set of vertices of a 3D model of a rock target. This represents the edge connectivity of the 3D model of the rock target. This represents a set of triangular facets representing a three-dimensional model of a rock target.

8. The method according to claim 1, characterized in that, This also includes geometric analysis of the three-dimensional model of the rock target, specifically: Obtain the spatial centroid of the rock target based on its 3D model: in: Let the geometric centroid of the rock target be... For the first One vertex, This represents the total number of vertices in the model. Establish the covariance matrix based on the spatial centroid of the rock target and the vertices of the 3D model of the rock target: Eigenvalue decomposition is performed on the covariance matrix to obtain three corresponding eigenvectors. The spatial attitude of the rock target is estimated by the three eigenvectors. The eigenvector corresponding to the largest eigenvalue is used as the principal direction of the rock target, and the other two eigenvectors together constitute the local coordinate system of the target. Based on the spatial attitude estimation of the rock target, the maximum and minimum coordinate values ​​of the three-dimensional model of the rock target in the three principal axis directions are obtained, and the spatial dimension parameters such as the length, width and height of the rock target are obtained. The minimum bounding box of the target is determined to estimate the size and volume of the rock target. Obtain the surface normal vectors corresponding to each triangular facet in the 3D model of the rock target based on the triangular mesh model.

9. A collaborative sensing and 3D reconstruction system for unstructured rock targets, characterized in that, include: The first 3D point cloud acquisition module is used to perform pixel point cloud mapping on the dual-view depth map and dual-view RGB map of the unstructured rock target area to obtain the dual-view 3D point cloud of the rock target. The second 3D point cloud acquisition module is used to construct an initial spatial variation matrix for the registration of the dual-view rock target 3D point cloud. The initial spatial variation matrix is ​​optimized by using an iterative nearest point method and a progressive spatial registration strategy to obtain an optimal spatial variation matrix. Based on the optimal spatial variation matrix, the dual-view rock target 3D point cloud is processed to obtain a spatially aligned unified rock target 3D point cloud. The 3D model acquisition module for rock targets is used to estimate the surface normal of the unified 3D point cloud of the rock target to obtain the surface normal vector. Based on the SPR surface reconstruction method, the relationship between the spatial position of the unified 3D point cloud of the rock target and the surface normal vector is established to obtain the 3D model of the rock target.

10. An electronic device, characterized in that, include: One or more processors; A memory for storing executable instructions of the processor, which, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-8.

Images