Method and device for registering a three-dimensional space based on a point cloud and a camera picture

Abstract

The application discloses a three-dimensional space registration method and device based on point cloud and camera pictures, relates to the technical field of data processing, and comprises the following steps: a virtual three-dimensional space is established, a camera picture is imported as a two-dimensional image, and a virtual camera component is initialized; geodetic coordinate system external parameter data set of a real camera is acquired, and the virtual camera component view is calibrated by using the geodetic coordinate system external parameter data set; virtual camera parameter information is extracted, and point cloud configuration parameters (an external parameter matrix and an internal parameter matrix) are generated; a laser point cloud file is imported as a three-dimensional point cloud, the point cloud view is calibrated according to the point cloud configuration parameters and the geodetic coordinate system, and a preliminary mapping relationship between the three-dimensional point cloud and the two-dimensional image is established; the three-dimensional point cloud is mapped to a two-dimensional image plane, a user interface is used to mark a feature point set in synchronization, the preliminary mapping relationship is adaptively optimized, and a space registration result is output. In this way, the technical effects of improving registration accuracy and registration efficiency and avoiding human errors are achieved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for three-dimensional spatial registration based on point clouds and camera images. Background Technology

[0002] Existing registration methods have several shortcomings. First, traditional registration processes often rely on manual point selection, where feature points are manually selected and matched between point clouds and images. This method is not only inefficient but also susceptible to human error, leading to unstable registration accuracy. Second, existing technologies often struggle to achieve high-precision registration when dealing with complex scenes, especially in establishing the correspondence between point cloud data and image data, where error accumulation is prone to occur. Summary of the Invention

[0003] This invention provides a three-dimensional spatial registration method and apparatus based on point clouds and camera images to solve the technical problems of low registration efficiency, limited accuracy, and strong dependence on human operation in the prior art, thereby achieving the technical effects of improving registration accuracy and efficiency and avoiding human error.

[0004] In a first aspect, the present invention provides a three-dimensional spatial registration method based on point clouds and camera images, wherein the method includes: Create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component.

[0005] Obtain the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and perform view calibration of the virtual camera component based on the extrinsic parameter dataset.

[0006] After interactive calibration, the virtual camera component extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes extrinsic parameter matrix and intrinsic parameter matrix.

[0007] Import the laser point cloud file as a 3D point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image by combining the calibrated virtual camera component.

[0008] The three-dimensional point cloud is mapped to a two-dimensional image plane based on the preliminary mapping relationship, and the feature marker point set is synchronized through the interactive user interface. Adaptive optimization of the preliminary mapping relationship is performed in combination with the mapping result, and spatial registration result is output.

[0009] Secondly, the present invention also provides a three-dimensional spatial registration device based on point clouds and camera images, wherein the device comprises: The space creation module is used to create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component.

[0010] The camera view calibration module is used to acquire the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and to perform view calibration of the virtual camera component based on the extrinsic parameter dataset.

[0011] The configuration parameter extraction module is used to extract virtual camera parameter information from the interactively calibrated virtual camera component and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes extrinsic parameter matrix and intrinsic parameter matrix.

[0012] The calibration and preliminary mapping module is used to import laser point cloud files as 3D point clouds, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image in conjunction with the calibrated virtual camera component.

[0013] The adaptive optimization module is used to map the 3D point cloud to a 2D image plane according to the preliminary mapping relationship, and synchronize the feature marker point set with the interactive user interface. It performs adaptive optimization of the preliminary mapping relationship in combination with the mapping result and outputs the spatial registration result.

[0014] This invention discloses a method and apparatus for three-dimensional spatial registration based on point clouds and camera images, comprising: constructing a virtual three-dimensional space, importing camera image files as two-dimensional images, and simultaneously initializing a virtual camera component; acquiring an external parameter dataset of a physical camera based on a geodetic coordinate system, and performing view calibration on the virtual camera component based on the external parameter dataset; interacting with the calibrated virtual camera component, extracting parameter information of the virtual camera, and generating point cloud configuration parameter information, including external parameter matrices and internal parameter matrices; importing a laser point cloud file as three-dimensional point cloud data, performing view calibration on the point cloud using the point cloud configuration parameter information and the geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and image points in the two-dimensional image based on the calibrated virtual camera component; based on the preliminary mapping relationship, projecting the three-dimensional point cloud onto the two-dimensional image plane, synchronizing the feature marker point set through the interactive user interface, and performing adaptive optimization of the preliminary mapping relationship based on the mapping result, outputting the final spatial registration result. The method and apparatus for three-dimensional spatial registration based on point clouds and camera images disclosed in this invention solve the technical problems of low registration efficiency, limited accuracy, and strong dependence on human operation, achieving the technical effects of improving registration accuracy and efficiency and avoiding human error. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the three-dimensional spatial registration method based on point cloud and camera image of the present invention. Figure 2This is a schematic diagram of the structure of the three-dimensional spatial registration device based on point cloud and camera image of the present invention.

[0016] Figure labeling: 11 Space establishment module, 12 Camera view calibration module, 13 Configuration parameter extraction module, 14 Calibration and preliminary mapping module, 15 Adaptive optimization module. Detailed Implementation

[0017] The technical solutions provided in the embodiments of the present invention address the technical problems of low registration efficiency, limited accuracy, and strong dependence on manual operation in the prior art. The overall approach adopted is as follows: First, camera image files are imported into the virtual 3D space as 2D images, and the virtual camera component is initialized. Then, the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system is obtained, and the virtual camera component is calibrated using this dataset. Next, the calibrated virtual camera component is interacted with to extract virtual camera parameter information, which is output as point cloud configuration parameter information, including extrinsic and intrinsic parameter matrices. Then, a laser point cloud file is imported as a 3D point cloud, and point cloud view calibration is performed based on the point cloud configuration parameter information and the geodetic coordinate system. Using the calibrated virtual camera component, a preliminary mapping relationship is established between the 3D point cloud and image points in the 2D image. Subsequently, the 3D point cloud is mapped to the 2D image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized in the user interface. Adaptive optimization of the preliminary mapping relationship is performed based on the mapping results, and finally, the spatial registration result is output.

[0018] The above technical solutions will now be described in detail with reference to the accompanying drawings and specific embodiments to provide a better understanding of them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0019] Example 1 Figure 1 This is a flowchart illustrating the three-dimensional spatial registration method based on point clouds and camera images according to the present invention, wherein the method includes: S100: Create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component.

[0020] Specifically, virtual 3D space refers to a 3D coordinate system or scene environment constructed using computer graphics technology to simulate the 3D geometric relationships of the real world, providing a unified reference framework for subsequent point cloud and image processing; camera image files refer to 2D image files taken by a camera, usually in .jpg or .png format, containing visual information of the scene.

[0021] Specifically, a virtual camera component refers to a software module that simulates the function of an actual camera in a virtual three-dimensional space. It can generate a view similar to that of an actual camera based on camera parameters (such as focal length, angle of view, position, etc.). Initializing the virtual camera component means setting its initial parameters so that it can be correctly aligned with the imported two-dimensional image. For example, this includes setting the initial position, initial height, initial angle of view, etc. of the virtual camera component.

[0022] By establishing a virtual 3D space and initializing a virtual camera component, unified management of point cloud and image data can be achieved, avoiding errors caused by inconsistencies in coordinate systems, which is common in traditional methods. Furthermore, the initialization of the virtual camera component allows for rapid alignment with a view that approximates that of the actual camera, reducing the time and workload of manually adjusting the view.

[0023] S200: Obtain the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and perform view calibration of the virtual camera component based on the extrinsic parameter dataset.

[0024] Specifically, the geodetic coordinate system is a coordinate system that uses the Earth as a reference and can accurately describe the position of an object on the ground. Through the geodetic coordinate system, a unified reference framework can be provided for the positioning of the camera. The extrinsic parameter dataset refers to the set of parameters that describe the position and orientation of the camera in three-dimensional space, including rotation matrices and translation vectors. In other words, the extrinsic parameter dataset defines the position and orientation of the physical camera relative to the reference coordinate system (geodetic coordinate system).

[0025] Specifically, by using an extrinsic dataset, the parameters of the virtual camera (such as position, orientation, focal length, etc.) are adjusted to ensure that the view generated by the virtual camera is geometrically consistent with the image captured by the actual camera, thus providing a high-precision initial alignment for subsequent point cloud and image registration.

[0026] In some embodiments, acquiring an extrinsic parameter dataset of a physical camera based on a geodetic coordinate system, and performing view calibration of the virtual camera component based on the extrinsic parameter dataset, includes: The camera control log of the interactive physical camera is used to extract the control G-code of the camera gimbal and determine the extrinsic parameter dataset. Based on the extrinsic parameter dataset, the pointing of the virtual camera component is adjusted, and the viewpoint, position, and focal length parameters of the virtual camera component are adjusted so that the viewpoint of the virtual camera is consistent with the viewpoint of the two-dimensional image.

[0027] Specifically, the camera control log refers to a log file that records the camera operation process. It usually contains motion control commands for the camera gimbal (such as G-code). Through this log, the position and attitude of the camera during the shooting process can be reflected. G-code is a programming language used to control CNC equipment (such as camera gimbals). It records the motion trajectory and operation commands of the equipment. By parsing G-code, the position and attitude information of the camera in three-dimensional space can be extracted.

[0028] Specifically, view calibration is the process of adjusting the parameters of a virtual camera (such as position, orientation, and focal length) to make its generated view geometrically consistent with the image captured by the actual camera. First, the control G-code is extracted by interacting with the camera control log of the physical camera, and the extrinsic parameter dataset of the physical camera is determined by analysis, including the camera's 3D position, orientation, and focal length parameters. Then, the parameters of the virtual camera component are adjusted according to this extrinsic parameter dataset to make its pointing, viewpoint, position, and focal length consistent with the physical camera.

[0029] For example, by combining the position of the fixed reference object configured when establishing the virtual three-dimensional space in the view, the virtual camera component is adjusted to achieve view calibration by minimizing the difference between the relative position of the fixed reference object in the virtual camera view and the relative position of the fixed reference object in the two-dimensional image. Here, the fixed reference object is an object in the scene that maintains a fixed position, such as ground markings, buildings or other stationary objects, and is used as a reference point for calibrating the virtual and real camera views.

[0030] Through the above process, the view of the virtual camera can be calibrated to the perspective of the two-dimensional image captured by the physical camera, thereby accurately reproducing the view effect of the physical camera in the virtual environment, avoiding errors caused by manual calibration, and improving operational efficiency and automation.

[0031] S300: The virtual camera component after interactive calibration extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes extrinsic parameter matrix and intrinsic parameter matrix.

[0032] Specifically, the point cloud configuration parameter information is a set of parameters used for subsequent point cloud processing and registration, including extrinsic and intrinsic parameter matrices. The extrinsic parameter matrix describes the position and attitude of the camera in the global coordinate system (such as the geodetic coordinate system), and usually includes rotation matrix and translation vector. The intrinsic parameter matrix describes the internal geometric characteristics of the camera (such as focal length, principal point position, pixel size, etc.), and reflects the camera's imaging model.

[0033] Specifically, the intrinsic and extrinsic parameter matrices of the virtual camera can be extracted from the view calibration results of the virtual camera components mentioned above. These matrices can be approximated as the intrinsic and extrinsic parameter matrices of the physical camera and used to guide the registration and mapping between the point cloud in the geodetic coordinate system and the two-dimensional image in the camera coordinate system.

[0034] Through the above process, the precise parameters of the virtual camera are used as point cloud configuration parameters and passed to the subsequent point cloud processing process. This provides key geometric information for the registration of point cloud and image data, ensuring that point cloud data and image data can be aligned with high precision in subsequent processing, thereby improving the accuracy and reliability of the entire registration process.

[0035] S400: Import the laser point cloud file as a 3D point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image by combining the calibrated virtual camera component.

[0036] Specifically, laser point cloud files are 3D point cloud data files acquired by a laser scanner, typically stored in .las or .ply format. Laser point cloud files contain the 3D coordinates (X, Y, Z) of a large number of spatial points, as well as possible additional information (such as intensity, color, etc.). The initial mapping relationship refers to the initial correspondence established between the point cloud data and the 2D image. This initial correspondence is represented by geometric transformations and projection algorithms, which can relatively accurately map the 3D points in the point cloud to the 2D image plane, providing a foundation for subsequent virtual-real overlay and precise matching.

[0037] Specifically, point cloud view calibration refers to the process of adjusting point cloud data to a geodetic coordinate system consistent with that of the virtual camera component, based on the coordinate information and configuration parameters (including extrinsic and intrinsic parameters) of the point cloud data.

[0038] In some embodiments, the imported laser point cloud file is a 3D point cloud. Point cloud view calibration is performed based on the point cloud configuration parameter information and the geodetic coordinate system. Then, in conjunction with the calibrated virtual camera component, a preliminary mapping relationship is established between the 3D point cloud and image points in the 2D image, including: Based on the coordinate information of the laser point cloud file, the laser point cloud file is imported into the virtual three-dimensional space as a three-dimensional point cloud; based on the point cloud configuration parameter information, the translation, rotation and affine transformation logic between the three-dimensional point cloud and the two-dimensional image is defined, and the output is the preliminary mapping relationship.

[0039] Specifically, firstly, the laser point cloud file (e.g., .las format file) is imported into a virtual 3D space and converted from a 3D point cloud coordinate system to 3D point cloud data in the virtual 3D space. Then, based on the point cloud configuration parameters (including extrinsic and intrinsic matrixes) and the geodetic coordinate system, the point cloud data is calibrated. This process uses geometric transformations such as translation, rotation, and scaling to adjust the point cloud data to a view consistent with the virtual camera component. Subsequently, using the calibrated virtual camera component, a projection algorithm is employed to project the 3D points in the point cloud onto a 2D image plane, thus establishing a preliminary mapping relationship between the point cloud and the image. This mapping relationship forms the basis for subsequent optimization and precise positioning.

[0040] Through the above process, importing the laser point cloud file and performing point cloud calibration using a geodetic coordinate system ensures that the 3D point cloud data accurately reproduces the position and structure of the real-world scene in virtual space. Based on the point cloud configuration parameter information, a preliminary mapping relationship involving translation, rotation, and affine transformation logic is defined, achieving preliminary geometric alignment between the point cloud data and the 2D image, providing initial conditions for subsequent optimization and precise positioning.

[0041] S500: Map the 3D point cloud to the 2D image plane according to the preliminary mapping relationship, synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output the spatial registration result.

[0042] Specifically, through the initial mapping relationship, the 3D point cloud imported into the virtual 3D space is mapped to the 2D image plane of the virtual camera component to realize the visualization of 3D data in the 2D view, and the initial mapping relationship is adaptively optimized based on the 2D plane space.

[0043] Specifically, a feature marker set consists of key points that are explicitly marked in a two-dimensional image or three-dimensional point cloud to represent specific geometric features or key locations, and is used to assist in alignment or calibration.

[0044] In some embodiments, the 3D point cloud is mapped to a 2D image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized through the interactive user interface. Adaptive optimization of the preliminary mapping relationship is then performed based on the mapping result, and a spatial registration result is output, including: The points in the 3D point cloud are projected onto the 2D image plane according to the preliminary mapping relationship to obtain a point cloud projection image. Based on the feature marker point set, a reference point set is determined in the 2D image, and a comparison point set is determined in the point cloud projection image. The positional association between the reference point set and the comparison point set is established. Combining an optimization algorithm, with the optimization of the multi-scale geometric features of the corresponding points in the reference point set and the comparison point set as the objective and the change of the reference point set as the penalty factor, the preliminary mapping relationship is iteratively optimized. When the iteratively optimized preliminary mapping relationship satisfies the preset multi-scale geometric feature objective or the preset number of optimization iterations, the positional difference of the feature marker point set before and after optimization is calculated based on the optimal optimization result, and the result is output as a 2D tuning parameter. Inverse mapping is performed based on the 2D tuning parameter to output the spatial registration result.

[0045] Specifically, the reference point set refers to the set of reference points determined based on the feature-marked point set in a two-dimensional image for registration; the comparison point set refers to the set of points in the point cloud projection image that correspond to the reference point set.

[0046] Specifically, multi-scale geometric features refer to geometric feature analysis used to evaluate point sets at different scales, such as minimizing the distance between corresponding points, minimizing the distance variance, and minimizing the distance mean; penalty factors refer to factors that constrain variables during the optimization process to limit the range of change during the optimization process.

[0047] Specifically, firstly, the points in the 3D point cloud are projected onto the 2D image plane according to a preliminary mapping relationship to generate a point cloud projection image; then, based on the feature marker point set obtained through user interface interaction, the reference point set in the 2D image and the comparison point set in the point cloud projection image are determined, and the point position association relationship between the two is established.

[0048] Next, the initial mapping relationship is adaptively optimized using an optimization algorithm. The goal of the optimization is to optimize the multi-scale geometric features (such as distance sum and distance variance) between the reference point set and the comparison point set. At the same time, the variation of the reference point set is configured as a penalty factor to limit the variation of the pixel position in the two-dimensional image, and only the spatial coordinate data of the three-dimensional point cloud is adjusted to complete the registration.

[0049] Specifically, when the optimized preliminary mapping relationship satisfies the preset geometric feature target or reaches the preset number of optimizations, the positional differences of multiple points in the feature marker point set (i.e., the comparison point set corresponding to the 3D point cloud) before and after optimization in the two-dimensional image plane are calculated, and the output is the two-dimensional tuning parameters. Then, according to the aforementioned preliminary mapping relationship, the two-dimensional tuning parameters are inversely mapped, that is, the registration adjustment parameters based on the two-dimensional scale of the image plane are transformed into the registration adjustment parameters in the 3D point cloud in the geodetic coordinate system through the aforementioned preliminary mapping relationship via affine transformation and translation and rotation, so as to perform adaptive optimization of the overall registration of the 3D point cloud and obtain the final spatial registration result.

[0050] The above process, through interactive optimization, further improves the registration accuracy between point clouds and 2D images, resolves errors in the initial mapping relationship, and ensures that point clouds and images are geometrically highly consistent. At the same time, registration optimization in the 2D plane helps improve the overall registration efficiency, further enhancing the practicality and adaptability of the registration method.

[0051] In some embodiments, the 3D point cloud is mapped to a 2D image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized through the interactive user interface. Adaptive optimization of the preliminary mapping relationship is then performed based on the mapping result, and a spatial registration result is output. Prior to this, the process includes: An interactive user interface is used to obtain point cloud sampling constraints, wherein the point cloud sampling constraints include a first downsampling parameter for key points and a second downsampling parameter for non-key points; An interactive user interface is used to obtain point cloud annotation information annotated by professional technicians, wherein the point cloud annotation information includes annotations of objects of interest and annotations of controlled objects; Based on the first downsampling parameter and the second downsampling parameter, downsampling processing is performed on the point cloud regions corresponding to the annotations of the object of interest and the annotations of the controlled object, respectively.

[0052] Specifically, point cloud sampling constraints refer to the parameter constraints on downsampling (i.e. reducing point cloud density) of point clouds to reduce data volume and improve computational efficiency when processing point cloud data. For example, controlling how many points to take a point is used to control the sparsity of the point cloud. The point cloud sampling constraints include a first downsampling parameter and a second downsampling parameter.

[0053] Specifically, the first downsampling parameter refers to the downsampling parameter for key points (such as guide lines, buildings, etc.) and is used to control the point cloud density of key areas; the second downsampling parameter refers to the downsampling parameter for non-key points (such as background, ground, etc.) and is used to control the point cloud density of non-key areas. Preferably, the first downsampling parameter is smaller than the second downsampling parameter, that is, the sampling interval under the first downsampling parameter is smaller than the sampling interval under the second downsampling parameter.

[0054] Specifically, point cloud annotation information refers to the annotations made by professional technicians in point clouds to distinguish different objects or regions, such as annotations of objects of interest (such as wires, buildings, etc.) and annotations of controlled objects.

[0055] Preferably, the aforementioned professional and technical personnel include natural human technicians and digital human technicians. Natural human technicians refer to human technicians with professional knowledge and experience who can provide accurate point cloud annotation information through manual annotation and analysis. Digital human technicians refer to virtual technicians realized through artificial intelligence or automated algorithms. These digital humans automatically identify and annotate key objects and controlled objects in point clouds based on preset rules (such as knowledge images, expert systems), machine learning models, or deep learning algorithms.

[0056] Specifically, firstly, point cloud sampling constraint parameters are obtained through an interactive user interface, including first downsampling parameters for key points and second downsampling parameters for non-key points. Key points (such as guide lines and buildings) typically require higher sampling density to preserve details, while non-key points (such as ground and background) can reduce data volume through larger downsampling intervals. Secondly, point cloud annotation information annotated by professional technicians is obtained through the user interface, including annotations of objects of interest (such as guide lines and buildings) and annotations of controlled objects (such as ground and background). This annotation information is used to distinguish different regions in the point cloud so that different regions can be processed in a targeted manner.

[0057] Furthermore, based on the acquired first and second downsampling parameters, downsampling processing is performed on the point cloud regions corresponding to the annotations of the objects of interest and the controlled objects, respectively, in order to reduce the amount of point cloud data, optimize the structure of the point cloud, and provide a more efficient data foundation for subsequent registration and optimization.

[0058] The purpose of the above process is to optimize the structure of point cloud data, reduce the amount of data, and improve the efficiency and accuracy of subsequent processing. In particular, through annotation by professional technicians and targeted downsampling, key information can be better preserved while redundant data is removed, providing high-quality input data for subsequent registration and optimization.

[0059] In some embodiments, the method further includes: The system identifies regions of interest (ROIs) from the 2D image and adaptively divides the data into 2D grids and grid images based on the identification results. Based on the 2D grids and spatial registration results, it performs adaptive slicing optimization on the 3D point cloud. The sliced ​​3D point cloud and grid images are stored in structured data, and an index file is created. A ranging request from the user interface is received, the pixel coordinates of the measuring points are extracted, and grid matching is performed between the pixel coordinates and the index file to locate the target segmented region. The system then retrieves the corresponding stored structured data as target point cloud data, combines the target point cloud data with the corresponding grid image to respond to the ranging request, and obtains the ranging result.

[0060] Specifically, region of interest (ROI) identification refers to identifying key regions in a two-dimensional image, such as wires, buildings, and hazard sources, through image processing algorithms (such as object detection and semantic segmentation). Adaptive meshing refers to automatically dividing a two-dimensional image into multiple meshes based on the distribution and characteristics of the ROI. Each mesh contains specific regional information. For example, when performing adaptive meshing on the ROI, the integrity and continuity of image features should be ensured as much as possible (e.g., based on image complexity or edge curvature).

[0061] Specifically, adaptive slicing optimization refers to dividing 3D point cloud data into multiple slices corresponding to the mesh based on the 2D mesh division and spatial registration results, optimizing the structure of the point cloud data to improve processing efficiency; structured data storage refers to storing the sliced ​​point cloud data and mesh images in a structured manner for easy access and processing.

[0062] Specifically, an index file is an index created for stored structured data to enable quick location and retrieval of target data.

[0063] Specifically, firstly, image processing algorithms are used to identify regions of interest (ROIs) in the 2D image, such as identifying key areas like conductors, buildings, and hazard sources. Then, based on the identification results, the 2D image is adaptively divided into multiple grids, each containing specific regional information, thus obtaining the 2D grid and grid image. Next, based on the 2D grid and spatial registration results, adaptive slicing optimization is performed on the 3D point cloud. This involves dividing the point cloud data into multiple slices corresponding to the grids, optimizing the structure of the point cloud data for easier subsequent processing. Finally, the sliced ​​point cloud data and grid image are stored in a structured manner, and an index file is created to facilitate rapid location and retrieval of target data.

[0064] Furthermore, it receives ranging requests from the user interface, extracts the pixel coordinates of the measuring points according to the ranging requests, and quickly locates the target segmented region through the index file; then, it calls the stored structured data to obtain the target point cloud data, and combines the target point cloud data with the corresponding grid image to respond to the ranging request and obtain the ranging result.

[0065] The above-described methods and steps, through structured storage and indexing mechanisms, improve data processing efficiency while providing users with fast and accurate ranging capabilities. Adaptive mesh partitioning and slicing optimization enable better handling of large-scale point cloud data, enhancing applicability and user experience.

[0066] In some embodiments, adaptive slicing optimization of the 3D point cloud based on the two-dimensional mesh and the spatial registration result further includes: The 3D point cloud after traversing the slices is used to calculate the shape feature parameters of each point based on the preset k-neighborhood parameters. The shape feature parameters are then determined based on the preset maximum allowable value of local shape. If the shape feature parameters are greater than or equal to the maximum allowable value of local shape, feature points in the corresponding region are extracted based on the first density. If the shape feature parameters are less than the maximum allowable value of local shape, key points in the corresponding region are extracted based on the second density. The feature points and key points are then fused, and duplicate data is removed to compress the point cloud.

[0067] Specifically, by further compressing the sliced ​​point cloud based on point extraction, the data volume and complexity of the 3D point cloud are effectively reduced, thereby improving the efficiency of analysis and storage.

[0068] Among them, the k-neighborhood parameter refers to the number k of the nearest points around each point that needs to be considered when calculating the shape features of each point in the point cloud. The choice of the value of k will affect the calculation accuracy and computational complexity of the shape features. The shape feature parameter refers to the parameter obtained by calculating the geometric relationship (such as curvature, normal vector consistency, etc.) between each point in the point cloud and its neighboring points, which is used to describe the local geometry of the point. The maximum allowable value of the local shape is a preset threshold used to judge whether the shape features of the point are significant. If the shape feature parameter exceeds the threshold, the point is considered to belong to the region with significant features.

[0069] Specifically, in the adaptive slicing optimization process of 3D point clouds, firstly, the sliced ​​point cloud data is traversed, and the shape feature parameters of each point are calculated. During the calculation, the geometric relationship between each point and its k neighboring points is considered, and the neighborhood range is determined by a preset k-neighborhood parameter. Subsequently, the shape feature parameters of each point are judged according to a preset maximum allowable value for local shape. If the shape feature parameter is greater than or equal to the maximum allowable value for local shape, the point is considered to belong to a region with significant features (such as edges, corners, etc.), and feature point extraction is performed based on a higher density (first density). If the shape feature parameter is less than the maximum allowable value for local shape, the point is considered to belong to a relatively flat or insignificant region (such as a point in a plane), which does not have significant geometric features but is structurally representative. In this case, key point extraction is performed based on a lower density (second density) to minimize the amount of point cloud data.

[0070] Finally, the extracted feature points and key points are fused and duplicate data is removed to compress the point cloud. This process not only preserves the important geometric information in the point cloud but also reduces the amount of data, providing a more efficient and concise data foundation for subsequent analysis and processing, and improving the efficiency of subsequent processing.

[0071] In summary, the 3D spatial registration method based on point cloud and camera image provided by this invention has the following technical effects: By constructing a virtual 3D space, importing camera image files as 2D images, and simultaneously initializing the virtual camera component; acquiring the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and performing view calibration on the virtual camera component based on this extrinsic parameter dataset; interacting with the calibrated virtual camera component, extracting the virtual camera's parameter information, and generating point cloud configuration parameter information, including extrinsic and intrinsic parameter matrices; importing laser point cloud files as 3D point cloud data, using the point cloud configuration parameter information and the geodetic coordinate system to perform view calibration on the point cloud, and combining the calibrated virtual camera component to establish a preliminary mapping relationship between the 3D point cloud and image points in the 2D image; based on the preliminary mapping relationship, projecting the 3D point cloud onto the 2D image plane, synchronously marking feature point sets in the interactive user interface, and performing adaptive optimization of the preliminary mapping relationship based on the mapping results, outputting the final spatial registration result, thereby achieving the technical effects of improving registration accuracy and efficiency and avoiding human error.

[0072] Example 2 Figure 2 This is a schematic diagram of the structure of the three-dimensional spatial registration device based on point clouds and camera images according to the present invention. For example, Figure 1 The flowchart of the three-dimensional spatial registration method based on point cloud and camera image of the present invention can be illustrated as follows: Figure 2 The structure shown is implemented.

[0073] Based on the same concept as the three-dimensional spatial registration method based on point cloud and camera image in the above embodiments, the present invention also provides a three-dimensional spatial registration device based on point cloud and camera image, comprising: The space creation module 11 is used to create a virtual three-dimensional space, import camera image files as two-dimensional images, and initialize the virtual camera component.

[0074] The camera view calibration module 12 is used to acquire the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and to perform view calibration of the virtual camera component based on the extrinsic parameter dataset.

[0075] The configuration parameter extraction module 13 is used to extract virtual camera parameter information from the calibrated virtual camera component and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix.

[0076] The calibration and preliminary mapping module 14 is used to import the laser point cloud file as a three-dimensional point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image in combination with the calibrated virtual camera component.

[0077] The adaptive optimization module 15 is used to map the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, and synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output the spatial registration result.

[0078] In some embodiments, the camera view calibration module 12 includes: The extrinsic parameter dataset determination unit is used to extract the camera control logs of the interactive physical camera, extract the control G-code of the camera gimbal, and determine the extrinsic parameter dataset.

[0079] The virtual camera view adjustment unit is used to adjust the pointing of the virtual camera component according to the extrinsic parameter dataset, and adjust the viewpoint, position and focal length parameters of the virtual camera component so that the viewpoint of the virtual camera is consistent with the viewpoint of the two-dimensional image.

[0080] In some embodiments, the calibration and preliminary mapping module 14 includes: The 3D point cloud import unit is used to import the laser point cloud file into a 3D point cloud in the virtual 3D space based on the coordinate information of the laser point cloud file.

[0081] The preliminary mapping relationship generation unit is used to define the translation, rotation and affine transformation logic between the three-dimensional point cloud and the two-dimensional image based on the point cloud configuration parameter information, and output the preliminary mapping relationship.

[0082] In some embodiments, the adaptive optimization module 15 includes: The point cloud projection image acquisition unit is used to project the points in the three-dimensional point cloud onto the two-dimensional image plane according to the preliminary mapping relationship, and acquire the point cloud projection image.

[0083] The point-location association establishment unit is used to determine a reference point set in the two-dimensional image and a comparison point set in the point cloud projection image based on the feature marker point set, and to establish the point-location association between the reference point set and the comparison point set.

[0084] The preliminary mapping relationship optimization unit is used to combine an optimization algorithm to iteratively optimize the preliminary mapping relationship by taking the optimal multi-scale geometric features of the corresponding points in the reference point set and the comparison point set as the objective and the change of the reference point set as the penalty factor.

[0085] The two-dimensional optimization parameter calculation unit is used to calculate the positional difference of the feature marker point set before and after optimization based on the optimal optimization result when the preliminary mapping relationship after iterative optimization satisfies the preset multi-scale geometric feature target or the preset number of optimization iterations, and outputs the two-dimensional optimization parameters.

[0086] The spatial registration result output unit is used to perform inverse mapping based on the two-dimensional tuning parameters and output the spatial registration result.

[0087] In some embodiments, the apparatus further includes a downsampling unit, configured to: use an interactive user interface to acquire point cloud sampling constraints, wherein the point cloud sampling constraints include a first downsampling parameter for key points and a second downsampling parameter for non-key points; use an interactive user interface to acquire point cloud annotation information annotated by professional technicians, wherein the point cloud annotation information includes annotations of objects of interest and annotations of controlled objects; and perform downsampling processing on the point cloud regions corresponding to the annotations of objects of interest and the annotations of controlled objects based on the first downsampling parameter and the second downsampling parameter, respectively.

[0088] In some implementations, the device further includes a slicing and secondary indexing unit, used for: identifying regions of interest based on the two-dimensional image, and adaptively dividing the image into grids based on the identification results to obtain a two-dimensional grid and a grid image; adaptively slicing and optimizing the three-dimensional point cloud based on the two-dimensional grid and the spatial registration result; storing the sliced ​​three-dimensional point cloud and the grid image in structured data and establishing an index file; receiving a ranging request from the user interface, extracting the pixel coordinates of the measuring points, and performing grid matching based on the pixel coordinates of the measuring points and the index file to locate the target slicing region; correspondingly calling the stored structured data as target point cloud data, and responding to the ranging request by combining the target point cloud data with the corresponding grid image to obtain the ranging result.

[0089] Furthermore, the slicing and secondary indexing unit is also used for: traversing the sliced ​​3D point cloud, calculating the shape feature parameters of each point according to the preset k-neighborhood parameters; determining the shape feature parameters according to the preset maximum allowable value of local shape; if the shape feature parameters are greater than or equal to the maximum allowable value of local shape, then extracting feature points of the corresponding region based on the first density; if the shape feature parameters are less than the maximum allowable value of local shape, then extracting key points of the corresponding region based on the second density; fusing the feature points and the key points, and deleting duplicate data to achieve point cloud compression.

[0090] It should be understood that the focus of the embodiments mentioned in this specification is their difference from other embodiments. The specific embodiments in the aforementioned Embodiment 1 are also applicable to the three-dimensional spatial registration device based on point cloud and camera image described in Embodiment 2. For the sake of brevity, they will not be further elaborated here.

[0091] It should be understood that the embodiments disclosed in this invention and the above description enable those skilled in the art to implement this invention. However, this invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention.

Claims

1. A three-dimensional spatial registration method based on point cloud and camera image, characterized in that, The method includes: Create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component; Obtain the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and perform view calibration of the virtual camera component based on the extrinsic parameter dataset; After interactive calibration, the virtual camera component extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix. Import the laser point cloud file as a 3D point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image by combining the calibrated virtual camera component. Based on the preliminary mapping relationship, the 3D point cloud is mapped to a 2D image plane, and the feature marker point set is synchronized through the interactive user interface. Adaptive optimization of the preliminary mapping relationship is then performed based on the mapping result, outputting spatial registration results, including: The points in the three-dimensional point cloud are projected onto the two-dimensional image plane according to the preliminary mapping relationship to obtain a point cloud projection image; Based on the feature marker point set, a reference point set is determined in the two-dimensional image, a comparison point set is determined in the point cloud projection image, and a point position association relationship is established between the reference point set and the comparison point set. Here, the reference point set refers to the set of reference points for registration determined based on the feature marker point set in the two-dimensional image. Combining optimization algorithms, with the goal of optimizing the multi-scale geometric features of the corresponding points in the reference point set and the comparison point set, and using the change in the reference point set as a penalty factor, the initial mapping relationship is iteratively optimized; When the preliminary mapping relationship after iterative optimization satisfies the preset multi-scale geometric feature target or the preset number of optimization iterations, the positional difference of the feature marker point set before and after optimization is calculated based on the optimal optimization result, and the output is a two-dimensional tuning parameter; Inverse mapping is performed based on the two-dimensional tuning parameters to output the spatial registration result.

2. The three-dimensional spatial registration method based on point cloud and camera image as described in claim 1, characterized in that, Obtaining the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and performing view calibration of the virtual camera component based on the extrinsic parameter dataset, including: The camera control logs of the interactive physical camera are used to extract the control G-code of the camera gimbal and determine the external parameter dataset. The virtual camera component is pointed according to the extrinsic parameter dataset, and the viewpoint, position and focal length parameters of the virtual camera component are adjusted so that the viewpoint of the virtual camera is consistent with the viewpoint of the two-dimensional image.

3. The three-dimensional spatial registration method based on point cloud and camera image as described in claim 2, characterized in that, Importing the laser point cloud file as a 3D point cloud, calibrating the point cloud view based on the point cloud configuration parameters and the geodetic coordinate system, and combining the calibrated virtual camera component, establishing a preliminary mapping relationship between the 3D point cloud and image points in the 2D image, including: Based on the coordinate information of the laser point cloud file, the laser point cloud file is imported into a three-dimensional point cloud in the virtual three-dimensional space; Based on the point cloud configuration parameter information, the translation, rotation and affine transformation logic of the 3D point cloud and the 2D image is defined, and the output is the preliminary mapping relationship.

4. The three-dimensional spatial registration method based on point cloud and camera image as described in claim 1, characterized in that, Based on the preliminary mapping relationship, the 3D point cloud is mapped to a 2D image plane, and the feature marker point set is synchronized through the interactive user interface. Adaptive optimization of the preliminary mapping relationship is then performed based on the mapping result, and the spatial registration result is output. Prior to this, the process includes: An interactive user interface is used to obtain point cloud sampling constraints, wherein the point cloud sampling constraints include a first downsampling parameter for key points and a second downsampling parameter for non-key points; An interactive user interface is used to obtain point cloud annotation information annotated by professional technicians, wherein the point cloud annotation information includes annotations of objects of interest and annotations of controlled objects; Based on the first downsampling parameter and the second downsampling parameter, downsampling processing is performed on the point cloud regions corresponding to the annotations of the object of interest and the annotations of the controlled object, respectively.

5. The three-dimensional spatial registration method based on point cloud and camera image as described in claim 1, characterized in that, The method further includes: Based on the two-dimensional image, the region of interest is identified, and adaptive grid division is performed in combination with the identification results to obtain a two-dimensional grid and a grid image. Based on the two-dimensional mesh division and the spatial registration result, the three-dimensional point cloud is subjected to adaptive slicing optimization; The sliced ​​3D point cloud and the mesh image are stored in a structured format, and an index file is created. Receive a ranging request from the user interface, extract the pixel coordinates of the measuring points, and perform grid matching between the pixel coordinates of the measuring points and the index file to locate the target segmented region; The corresponding stored structured data is the target point cloud data. The target point cloud data is combined with the corresponding grid image to make a ranging request response and obtain the ranging result.

6. The three-dimensional spatial registration method based on point cloud and camera image as described in claim 5, characterized in that, Based on the two-dimensional mesh division and the spatial registration result, adaptive slicing optimization of the three-dimensional point cloud is performed, which also includes: The shape feature parameters of each point are calculated based on the preset k-neighborhood parameters after traversing the sliced ​​3D point cloud. The shape feature parameters are determined based on the preset maximum allowable value of the local shape; If the shape feature parameter is greater than or equal to the maximum allowable value of the local shape, then feature points of the corresponding region are extracted based on the first density; If the shape feature parameter is less than the maximum allowable value of the local shape, then key points in the corresponding region are extracted based on the second density; The point cloud is compressed by fusing the feature points and the key points and removing duplicate data.

7. A three-dimensional spatial registration device based on point clouds and camera images, characterized in that, The apparatus is used to perform the three-dimensional spatial registration method based on point cloud and camera image as described in any one of claims 1-6, and the apparatus comprises: The space creation module is used to create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component; The camera view calibration module is used to acquire the extrinsic parameter dataset of the physical camera based on the geodetic coordinate system, and to perform view calibration of the virtual camera component based on the extrinsic parameter dataset. The configuration parameter extraction module is used to interactively calibrate the virtual camera component, extract virtual camera parameter information and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix; The calibration and preliminary mapping module is used to import laser point cloud files as 3D point clouds, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image in combination with the calibrated virtual camera component. An adaptive optimization module is used to map the 3D point cloud to a 2D image plane according to the preliminary mapping relationship, and synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output spatial registration result; The adaptive optimization module includes: A point cloud projection image acquisition unit is used to project the points in the three-dimensional point cloud onto the two-dimensional image plane according to the preliminary mapping relationship, and acquire a point cloud projection image. The point-position association establishment unit is used to determine a reference point set in the two-dimensional image and a comparison point set in the point cloud projection image based on the feature marker point set, and to establish the point-position association between the reference point set and the comparison point set. The reference point set refers to the set of reference points for registration determined based on the feature marker point set in the two-dimensional image. The preliminary mapping relationship optimization unit is used to combine an optimization algorithm to iteratively optimize the preliminary mapping relationship by taking the optimal multi-scale geometric features of the corresponding points in the reference point set and the comparison point set as the objective and the change of the reference point set as the penalty factor. The two-dimensional optimization parameter calculation unit is used to calculate the positional difference of the feature marker point set before and after optimization based on the optimal optimization result when the preliminary mapping relationship after iterative optimization satisfies the preset multi-scale geometric feature target or the preset number of optimization iterations, and outputs the two-dimensional optimization parameters. The spatial registration result output unit is used to perform inverse mapping based on the two-dimensional tuning parameters and output the spatial registration result.

Images