Three-dimensional space registration method and device based on point cloud and camera picture

Abstract

The invention discloses a three-dimensional space registration method and device based on a point cloud and a camera picture, and relates to the technical field of data processing, and the method comprises the steps: building a virtual three-dimensional space, importing the camera picture as a two-dimensional image, and initializing a virtual camera assembly; acquiring a geodetic coordinate system external parameter data set of the physical camera, and calibrating the virtual camera assembly view by using the geodetic coordinate system external parameter data set; virtual camera parameter information is extracted, and point cloud configuration parameters (an external parameter matrix and an internal parameter matrix) are generated; a laser point cloud file is imported as a three-dimensional point cloud, a point cloud view is calibrated according to point cloud configuration parameters and a geodetic coordinate system, and a preliminary mapping relation between the three-dimensional point cloud and a two-dimensional image is established; and mapping the three-dimensional point cloud to a two-dimensional image plane, synchronizing the feature mark point set through a user interface, performing adaptive optimization on the preliminary mapping relation, and outputting a spatial registration result. Therefore, the technical effects of improving the registration precision and the registration efficiency and avoiding personal errors are achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a three-dimensional space registration method and device based on point clouds and camera images. Background Art

[0002] Existing registration methods have numerous shortcomings. First, the traditional registration process often relies on manual point-puncturing, where feature points are manually selected and matched between the point cloud and the image. This method is not only inefficient but also susceptible to human factors, resulting in unstable registration accuracy. Second, existing technologies often struggle to achieve high-precision registration when processing complex scenes. This is particularly true when establishing the correspondence between point cloud data and image data, which can lead to error accumulation. Summary of the Invention

[0003] The present invention provides a three-dimensional space registration method and device based on point clouds and camera images to solve the technical problems of low registration efficiency, limited accuracy and strong dependence on human operation in the existing technology, and achieve the technical effect of improving registration accuracy and efficiency and avoiding human errors.

[0004] In a first aspect, the present invention provides a three-dimensional space registration method based on point clouds and camera images, wherein the method comprises: Create a virtual 3D space, import camera image files as 2D images, and initialize the virtual camera component.

[0005] An external parameter data set of a physical camera based on a geodetic coordinate system is obtained, and a view calibration of the virtual camera assembly is performed according to the external parameter data set.

[0006] After interactive calibration, the virtual camera component extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix.

[0007] Import the laser point cloud file as a three-dimensional point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image in combination with the calibrated virtual camera component.

[0008] The three-dimensional point cloud is mapped to a two-dimensional image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized through an interactive user interface. The preliminary mapping relationship is adaptively optimized in combination with the mapping result, and a spatial registration result is output.

[0009] In a second aspect, the present invention further provides a three-dimensional space registration device based on point clouds and camera images, wherein the device comprises: The space establishment module is used to establish a virtual three-dimensional space, import camera image files as two-dimensional images, and initialize the virtual camera component.

[0010] The camera view calibration module is used to obtain an external parameter data set of a physical camera based on a geodetic coordinate system, and perform view calibration of the virtual camera assembly according to the external parameter data set.

[0011] The configuration parameter extraction module is used to extract virtual camera parameter information from the interactively calibrated virtual camera component and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an external parameter matrix and an internal parameter matrix.

[0012] The calibration and preliminary mapping module is used to import the laser point cloud file as a three-dimensional point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image in combination with the calibrated virtual camera component.

[0013] An adaptive optimization module is used to map the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output the spatial alignment result.

[0014] The present invention discloses a three-dimensional spatial registration method and device based on point clouds and camera images, comprising: constructing a virtual three-dimensional space, importing a camera image file as a two-dimensional image, and simultaneously initializing a virtual camera component; obtaining an external parameter data set of a physical camera based on a geodetic coordinate system, and performing view calibration on the virtual camera component based on the external parameter data set; interacting with the virtual camera component after view calibration, extracting parameter information of the virtual camera, and generating point cloud configuration parameter information, including an external parameter matrix and an internal parameter matrix; importing a laser point cloud file as three-dimensional point cloud data, performing view calibration on the point cloud using the point cloud configuration parameter information and a geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and image points in the two-dimensional image in combination with the calibrated virtual camera component; based on the preliminary mapping relationship, projecting the three-dimensional point cloud onto a two-dimensional image plane, synchronizing a feature marker point set through an interactive user interface, and performing adaptive optimization of the preliminary mapping relationship based on the mapping result, and outputting a final spatial registration result. The three-dimensional spatial registration method and device based on point clouds and camera images disclosed in the present invention solve the technical problems of low registration efficiency, limited accuracy, and strong dependence on human operation, thereby achieving the technical effects of improving registration accuracy and efficiency and avoiding human errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the process of the three-dimensional space registration method based on point cloud and camera image of the present invention; Figure 2Schematic diagram of the structure of the three-dimensional space registration device based on point cloud and camera image of the present invention.

[0016] Description of reference numerals: space establishment module 11 , camera view calibration module 12 , configuration parameter extraction module 13 , calibration and preliminary mapping module 14 , adaptive optimization module 15 . DETAILED DESCRIPTION

[0017] The technical solutions provided in the embodiments of the present invention are designed to address the technical problems of low registration efficiency, limited accuracy, and strong reliance on manual operation in the prior art. The overall approach adopted is as follows: First, a camera image file is imported as a 2D image into the virtual 3D space, and the virtual camera component is initialized. Next, an extrinsic parameter dataset of the physical camera based on the earth coordinate system is obtained, and the virtual camera component is calibrated using this extrinsic parameter dataset. The interactively calibrated virtual camera component then extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, where the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix. Next, a laser point cloud file is imported as a 3D point cloud. Point cloud view calibration is performed based on the point cloud configuration parameter information and the earth coordinate system. Combined with the calibrated virtual camera component, a preliminary mapping relationship between the 3D point cloud and the image points in the 2D image is established. Furthermore, based on this preliminary mapping relationship, the 3D point cloud is mapped to the 2D image plane, and the feature marker point set is synchronized in the user interface. Adaptive optimization of the preliminary mapping relationship is performed based on the mapping results, and finally, the spatial registration result is output.

[0018] The above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification to better understand the above technical solution. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, it should be noted that, for the convenience of description, only the parts related to the present invention, rather than all, are shown in the drawings.

[0019] Example 1 Figure 1 The figure is a flow chart of a three-dimensional space registration method based on point clouds and camera images according to the present invention, wherein the method comprises: S100: Establish a virtual three-dimensional space, import a camera image file as a two-dimensional image, and initialize a virtual camera component.

[0020] Specifically, virtual three-dimensional space refers to a three-dimensional coordinate system or scene environment constructed through computer graphics technology, which is used to simulate the three-dimensional geometric relationships of the real world and provide a unified reference framework for subsequent point cloud and image processing; camera image files refer to two-dimensional image files taken by a camera, usually in .jpg or .png format, which contain visual information of the scene.

[0021] Specifically, a virtual camera component refers to a software module that simulates the functions of an actual camera in a virtual three-dimensional space. It can generate a view similar to that of an actual camera based on the camera parameters (such as focal length, viewing angle, position, etc.). Initializing the virtual camera component refers to setting its initial parameters so that it can be correctly aligned with the imported two-dimensional image. Exemplarily, it includes setting the initial position, initial height, initial viewing angle, etc. of the virtual camera component.

[0022] By establishing a virtual 3D space and initializing a virtual camera component, we can achieve unified management of point cloud and image data, avoiding the errors caused by inconsistent coordinate systems in traditional methods. Furthermore, initializing the virtual camera component allows for rapid alignment of views that approximate those of the actual camera, reducing the time and effort required for manual view adjustments.

[0023] S200: Acquire an extrinsic parameter data set of a physical camera based on a geodetic coordinate system, and perform view calibration of the virtual camera assembly according to the extrinsic parameter data set.

[0024] Specifically, the geodetic coordinate system is a coordinate system with the earth as reference, which can accurately describe the position of objects on the ground. Through the geodetic coordinate system, a unified reference framework can be provided for camera positioning; the extrinsic parameter data set refers to a set of parameters that describe the position and posture of the camera in three-dimensional space, including the rotation matrix and translation vector. In other words, the extrinsic parameter data set defines the position and orientation of the physical camera relative to the reference coordinate system (geodotoid coordinate system).

[0025] Specifically, the parameters of the virtual camera (such as position, orientation, focal length, etc.) are adjusted through the external reference data set, so that the view it generates is geometrically consistent with the image taken by the actual camera, providing high-precision initial alignment for subsequent point cloud and image registration.

[0026] In some embodiments, obtaining an external parameter dataset of a physical camera based on a geodetic coordinate system, and performing view calibration of the virtual camera assembly according to the external parameter dataset, includes: The camera control log of the interactive physical camera is used to extract the control G-code of the camera gimbal to determine the external reference data set; the pointing direction of the virtual camera component is adjusted according to the external reference data set, and the viewing angle, position and focal length parameters of the virtual camera component are adjusted to make the viewing angle of the virtual camera consistent with the viewing angle of the two-dimensional image.

[0027] Specifically, a camera control log refers to a log file that records the camera's operation process, usually containing motion control instructions for the camera gimbal (such as G-code). This log can reflect the camera's position and posture during the shooting process. G-code is a programming language used to control CNC equipment (such as camera gimbals), which records the equipment's motion trajectory and operation instructions. By parsing G-code, the camera's position and posture information in three-dimensional space can be extracted.

[0028] Specifically, view calibration is the process of adjusting the parameters of a virtual camera (such as position, orientation, and focal length) so that the view it generates is geometrically consistent with the image captured by the actual camera. First, the control G-code is extracted by interacting with the camera control log of the physical camera, and the extrinsic parameter data set of the physical camera is determined through analysis, including the camera's three-dimensional position, orientation, and focal length parameters. Then, the parameters of the virtual camera component are adjusted based on this extrinsic parameter data set so that its pointing, viewing angle, position, and focal length are consistent with those of the physical camera.

[0029] Exemplarily, in combination with the position of the fixed reference object configured in the view when establishing the virtual three-dimensional space, the direction of the virtual camera component is adjusted by minimizing the difference between the relative position of the fixed reference object in the virtual camera perspective and the relative position of the fixed reference object in the two-dimensional image, thereby achieving view calibration, wherein the fixed reference object is an object that maintains a fixed position in the scene, such as a ground sign, a building or other stationary object, which is used as a reference point for calibrating the virtual and real camera views.

[0030] Through the above process, the view of the virtual camera can be calibrated to the two-dimensional image perspective taken by the real camera, so as to accurately reproduce the view effect of the real camera in the virtual environment, avoid the errors caused by manual calibration, and improve operational efficiency and automation.

[0031] S300: extracting virtual camera parameter information from the interactively calibrated virtual camera component and outputting the information as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix.

[0032] Specifically, the point cloud configuration parameter information is a set of parameters used for subsequent point cloud processing and registration, including an extrinsic parameter matrix and an intrinsic parameter matrix. The extrinsic parameter matrix is ​​a matrix that describes the position and posture of the camera in a global coordinate system (such as the earth coordinate system), usually including a rotation matrix and a translation vector; the intrinsic parameter matrix is ​​a matrix that describes the internal geometric characteristics of the camera (such as focal length, principal point position, pixel size, etc.), reflecting the imaging model of the camera.

[0033] Specifically, through the view calibration results of the above-mentioned virtual camera components, the intrinsic parameter matrix and extrinsic parameter matrix of the virtual camera can be extracted, which can be approximately regarded as the intrinsic parameter matrix and extrinsic parameter matrix of the real camera, and are used to guide the alignment and mapping between the point cloud in the geodetic coordinate system and the two-dimensional image in the camera coordinate system.

[0034] Through the above process, the precise parameters of the virtual camera are passed as point cloud configuration parameter information to the subsequent point cloud processing process, providing key geometric information for the registration of point cloud and image, so as to ensure that the point cloud data and image data can be aligned with high precision in subsequent processing, thereby improving the accuracy and reliability of the entire registration process.

[0035] S400: Importing a laser point cloud file as a three-dimensional point cloud, calibrating the point cloud view based on the point cloud configuration parameter information and the geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image in combination with the calibrated virtual camera component.

[0036] Specifically, a laser point cloud file is a 3D point cloud data file acquired by a laser scanner, typically stored in .las or .ply formats. A laser point cloud file contains the 3D coordinates (X, Y, Z) of a large number of spatial points, as well as possible additional information (such as intensity and color). A preliminary mapping relationship is the initial correspondence established between the point cloud data and the 2D image. This initial correspondence, expressed as a geometric transformation and projection algorithm, can relatively accurately map the 3D points in the point cloud to the 2D image plane, providing the basis for subsequent virtual-real overlay and precise matching.

[0037] Specifically, point cloud view calibration refers to the process of adjusting point cloud data to a geodetic coordinate system consistent with the virtual camera component based on the coordinate information of the point cloud data and the point cloud configuration parameter information (including the extrinsic parameter matrix and the intrinsic parameter matrix).

[0038] In some embodiments, the laser point cloud file is imported as a three-dimensional point cloud, a point cloud view is calibrated based on the point cloud configuration parameter information and the geodetic coordinate system, and a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image is established in combination with the calibrated virtual camera component, including: According to the coordinate information of the laser point cloud file, the laser point cloud file is imported into the three-dimensional point cloud in the virtual three-dimensional space; based on the point cloud configuration parameter information, the translation, rotation and affine transformation logic of the three-dimensional point cloud and the two-dimensional image are defined, and the preliminary mapping relationship is output.

[0039] Specifically, a laser point cloud file (such as a .las format file) is first imported into a virtual 3D space and converted from a 3D point cloud coordinate system into 3D point cloud data in the virtual 3D space. Then, based on the point cloud configuration parameter information (including the extrinsic and intrinsic parameter matrices) and the earth coordinate system, the point cloud data is calibrated. This process adjusts the point cloud data to a view consistent with the virtual camera assembly through geometric transformations such as translation, rotation, and scaling. Subsequently, using the calibrated virtual camera assembly, a projection algorithm is used to project the 3D points in the point cloud onto a 2D image plane, thereby establishing a preliminary mapping relationship between the point cloud and the image. This mapping relationship forms the basis for subsequent optimization and precise positioning.

[0040] Through the above process, importing the laser point cloud file and calibrating the point cloud in conjunction with the geodetic coordinate system ensures that the 3D point cloud data accurately reproduces the position and structure of the real scene in virtual space. Based on the point cloud configuration parameters, a preliminary mapping relationship involving translation, rotation, and affine transformation logic is defined, achieving preliminary geometric alignment between the point cloud data and the 2D image, providing the initial conditions for subsequent optimization and precise positioning.

[0041] S500: Mapping the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, synchronizing a feature marker point set through an interactive user interface, performing adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and outputting a spatial registration result.

[0042] Specifically, through the preliminary mapping relationship, the three-dimensional point cloud imported into the virtual three-dimensional space is mapped to the two-dimensional image plane based on the virtual camera component to realize the visualization of the three-dimensional data in the two-dimensional view, and the preliminary mapping relationship is adaptively optimized based on the two-dimensional plane space.

[0043] Specifically, the feature marker point set is a key point that is explicitly marked in a two-dimensional image or a three-dimensional point cloud, which is used to represent specific geometric features or key positions and is used to assist alignment or calibration.

[0044] In some embodiments, the three-dimensional point cloud is mapped to a two-dimensional image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized with an interactive user interface. Adaptive optimization of the preliminary mapping relationship is performed in combination with the mapping result, and a spatial registration result is output, including: The points in the three-dimensional point cloud are projected onto the two-dimensional image plane according to the preliminary mapping relationship to obtain a point cloud projection image; based on the feature marker point set, a reference point set is determined in the two-dimensional image, and a comparison point set is determined in the point cloud projection image, and a point position association relationship between the reference point set and the comparison point set is established; combined with an optimization algorithm, the preliminary mapping relationship is iteratively optimized with the goal of optimizing the multi-scale geometric features of the corresponding points in the reference point set and the comparison point set, and the change of the reference point set is used as a penalty factor; when the iteratively optimized preliminary mapping relationship meets the preset multi-scale geometric feature goal or the preset number of iterative optimizations, the position difference of the feature marker point set before and after optimization is calculated based on the optimal optimization result, and output as a two-dimensional tuning parameter; inverse mapping is performed according to the two-dimensional tuning parameter, and a spatial alignment result is output.

[0045] Specifically, the reference point set refers to a set of reference points used for registration determined based on a feature marker point set in a two-dimensional image; the comparison point set refers to a set of points in a point cloud projection image that corresponds to the reference point set.

[0046] Specifically, multi-scale geometric features refer to geometric feature analysis used to evaluate point sets at different scales, such as the minimum distance between corresponding points, the minimum distance variance, the minimum distance average, etc.; the penalty factor refers to the factor that constrains the variables during the optimization process, which is used to limit the range of change in the optimization process.

[0047] Specifically, first, the points in the three-dimensional point cloud are projected onto the two-dimensional image plane according to the preliminary mapping relationship to generate a point cloud projection image; then, based on the feature marker point set obtained through user interface interaction, the reference point set in the two-dimensional image and the comparison point set in the point cloud projection image are determined, and a point position association relationship between the two is established.

[0048] Next, the preliminary mapping relationship is adaptively optimized through an optimization algorithm. The optimization goal is to optimize the multi-scale geometric features (such as distance sum, distance variance, etc.) between the reference point set and the comparison point set. At the same time, the reference point set change is configured as a penalty factor to limit the change of the pixel position of the two-dimensional image, and only the spatial coordinate data of the three-dimensional point cloud is adjusted to complete the alignment.

[0049] Specifically, when the optimized preliminary mapping relationship meets the preset geometric feature target or reaches the preset number of optimization times, the position difference of multiple points in the feature marker point set before and after optimization (that is, the comparison point set corresponding to the three-dimensional point cloud) in the two-dimensional image plane is calculated and output as a two-dimensional tuning parameter; then, according to the aforementioned preliminary mapping relationship, the two-dimensional tuning parameters are inversely mapped, that is, the registration adjustment parameters based on the two-dimensional scale of the image plane are converted into registration adjustment parameters in the three-dimensional point cloud in the geodetic coordinate system through the aforementioned preliminary mapping relationship, through affine transformation and translation and rotation, so as to perform adaptive optimization of the registration of the three-dimensional point cloud as a whole and obtain the final spatial registration result.

[0050] Through interactive optimization, the above process can further improve the registration accuracy between the point cloud and the two-dimensional image, resolve the errors in the preliminary mapping relationship, and ensure that the point cloud and the image are highly geometrically consistent. At the same time, the registration optimization in the two-dimensional plane helps to improve the overall registration efficiency, further enhancing the practicality and adaptability of the registration method.

[0051] In some embodiments, the three-dimensional point cloud is mapped to a two-dimensional image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized through an interactive user interface. Adaptive optimization of the preliminary mapping relationship is performed in combination with the mapping result, and a spatial registration result is output, before which the following steps are included: An interactive user interface is provided for obtaining a point cloud sampling constraint, wherein the point cloud sampling constraint includes a first downsampling parameter for a key point and a second downsampling parameter for a non-key point; An interactive user interface is used to obtain point cloud annotation information annotated by professional technicians, wherein the point cloud annotation information includes annotations of objects of interest and annotations of controlled objects; Based on the first downsampling parameter and the second downsampling parameter, downsampling processing is performed on the point cloud areas corresponding to the focus object label and the controlled object label respectively.

[0052] Specifically, point cloud sampling constraints refer to parameter constraints for downsampling point clouds (i.e., reducing point cloud density) to reduce data volume and improve computational efficiency when processing point cloud data. For example, this constraint controls the interval between points at which a point is taken, which is used to control the sparsity of the point cloud. The point cloud sampling constraints include a first downsampling parameter and a second downsampling parameter.

[0053] Specifically, the first downsampling parameter refers to the downsampling parameter for key points (such as wires, buildings, etc.), which is used to control the point cloud density in key areas; the second downsampling parameter refers to the downsampling parameter for non-key points (such as background, ground, etc.), which is used to control the point cloud density in non-key areas. Preferably, the first downsampling parameter is smaller than the second downsampling parameter, that is, the sampling interval under the first downsampling parameter is smaller than the sampling interval under the second downsampling parameter.

[0054] Specifically, point cloud annotation information refers to the annotations made by professional technicians in the point cloud, which are used to distinguish different objects or areas, such as the annotations of objects of interest (such as wires, buildings, etc.) and controlled objects.

[0055] Preferably, the above-mentioned professional and technical personnel include natural human technicians and digital human technicians. Natural human technicians refer to human technicians with professional knowledge and experience who can provide accurate point cloud annotation information through manual annotation and analysis; digital human technicians refer to virtual technicians realized by artificial intelligence or automated algorithms. These digital humans automatically identify and annotate key objects and controlled objects in the point cloud based on preset rules (such as knowledge images, expert systems), machine learning models or deep learning algorithms.

[0056] Specifically, first, the point cloud sampling constraint parameters are obtained through the interactive user interface, including the first downsampling parameters for key points and the second downsampling parameters for non-key points; among them, key points (such as wires, buildings, etc.) usually require a higher sampling density to retain details, while non-key points (such as the ground, background, etc.) can reduce the amount of data by a larger downsampling interval; then, the point cloud annotation information annotated by professional technicians is obtained through the user interface, including the annotation of objects of interest (such as wires, buildings, etc.) and the annotation of controlled objects (such as the ground, background, etc.); this annotation information is used to distinguish different areas in the point cloud so that targeted processing can be performed on different areas.

[0057] Furthermore, based on the obtained first downsampling parameters and second downsampling parameters, the point cloud areas corresponding to the focus object annotation and the controlled object annotation are downsampled respectively to reduce the amount of point cloud data, optimize the structure of the point cloud, and provide a more efficient data basis for subsequent alignment and optimization.

[0058] The purpose of the above process is to optimize the structure of point cloud data, reduce the amount of data, and improve the efficiency and accuracy of subsequent processing. Among them, through the annotation and targeted downsampling processing by professional technicians, key information can be better retained, while redundant data can be removed, providing high-quality input data for subsequent alignment and optimization.

[0059] In some embodiments, the method further comprises: Identify a region of interest based on the two-dimensional image, and adaptively perform grid division based on the identification result to obtain a two-dimensional divided grid and a grid image; adaptively slice and optimize the three-dimensional point cloud based on the two-dimensional divided grid and the spatial registration result; store structured data of the sliced ​​three-dimensional point cloud and the grid image, and create an index file; receive a ranging request from a user interface, extract the pixel coordinate position of the measuring point, and perform grid matching based on the pixel coordinate position of the measuring point and the index file to locate the target slice area; correspondingly call the stored structured data as target point cloud data, respond to the ranging request based on the target point cloud data and the corresponding grid image, and obtain a ranging result.

[0060] Specifically, region of interest recognition refers to identifying key areas in a two-dimensional image, such as wires, buildings, hazards, etc., through image processing algorithms (such as target detection, semantic segmentation, etc.); adaptive grid division refers to automatically dividing a two-dimensional image into multiple grids based on the distribution and characteristics of the region of interest, each grid contains specific area information. For example, when performing adaptive grid division on the region of interest, the integrity and continuity of the image features are ensured as much as possible (such as division based on image complexity or edge curvature).

[0061] Specifically, adaptive slicing optimization refers to dividing three-dimensional point cloud data into multiple slices corresponding to the grid based on the two-dimensional grid division and spatial alignment results, optimizing the structure of the point cloud data to improve processing efficiency; structured data storage refers to storing the sliced ​​point cloud data and grid images in a structured manner for fast access and processing.

[0062] Specifically, the index file is an index established for the stored structured data to facilitate rapid location and retrieval of target data.

[0063] Specifically, first, the image processing algorithm is used to identify the areas of interest in the two-dimensional image, such as key areas such as wires, buildings, and hazardous sources; then, based on the recognition results, the two-dimensional image is adaptively divided into multiple grids, each grid containing specific area information, thereby obtaining a two-dimensional divided grid and a grid image; next, based on the two-dimensional divided grid and spatial alignment results, the three-dimensional point cloud is adaptively sliced ​​and optimized, that is, the point cloud data is divided into multiple slices corresponding to the grid according to the two-dimensional divided grid, and the structure of the point cloud data is optimized for subsequent processing. Finally, the sliced ​​point cloud data and grid image are stored in a structured manner, and an index file is established to quickly locate and retrieve the target data.

[0064] Furthermore, a ranging request from the user interface is received, and according to the ranging request, the pixel coordinate position of the measuring point is correspondingly extracted, and the target segment area is quickly located through the index file; then, the stored structured data is called to obtain the target point cloud data, and the ranging request response is performed by combining the target point cloud data with the corresponding grid image to obtain the ranging result.

[0065] The above method improves data processing efficiency through structured storage and indexing mechanisms, while providing users with fast and accurate distance measurement capabilities. Adaptive meshing and slicing optimization can better handle large-scale point cloud data, improving applicability and user experience.

[0066] In some embodiments, performing adaptive slicing optimization on the three-dimensional point cloud based on the two-dimensional grid division and the spatial registration result further includes: The three-dimensional point cloud after slicing is traversed, and the shape feature parameters of each point are calculated according to the preset k-neighborhood parameters; the shape feature parameters are judged according to the preset local shape maximum allowable value; if the shape feature parameters are greater than or equal to the local shape maximum allowable value, feature points of the corresponding area are extracted based on the first density; if the shape feature parameters are less than the local shape maximum allowable value, key points of the corresponding area are extracted based on the second density; the feature points and the key points are fused, and duplicate data is deleted to achieve point cloud compression.

[0067] Specifically, by further performing point cloud compression based on point extraction on the sliced ​​point cloud, the data volume and complexity of the three-dimensional point cloud can be effectively reduced, and the efficiency of analysis and storage can be improved.

[0068] Among them, the k-neighborhood parameter refers to the number k of the nearest points around each point that needs to be considered when calculating the shape features of each point in the point cloud. The choice of k value will affect the calculation accuracy and complexity of the shape features; the shape feature parameter refers to the parameter obtained by calculating the geometric relationship between each point in the point cloud and its neighborhood points (such as curvature, normal vector consistency, etc.), which is used to describe the local geometric shape of the point; the maximum allowable value of the local shape is a preset threshold used to judge whether the shape features of the point are significant. If the shape feature parameter exceeds the threshold, the point is considered to belong to the area with significant features.

[0069] Specifically, in the process of adaptive slicing optimization of three-dimensional point clouds, first, the sliced ​​point cloud data is traversed, and the shape feature parameters of each point are calculated. During the calculation, the geometric relationship between each point and its surrounding k neighboring points is considered, and the neighborhood range is determined by the preset k neighborhood parameters. Subsequently, the shape feature parameters of each point are judged according to the preset local shape maximum allowable value. If the shape feature parameter is greater than or equal to the local shape maximum allowable value, the point is considered to belong to a feature-significant area (such as an edge, corner point, etc.). At this time, feature point extraction is performed based on a higher density (first density); if the shape feature parameter is less than the local shape maximum allowable value, the point is considered to belong to a relatively flat or non-significant area (such as a point in a plane), which does not have significant geometric features but is structurally representative. At this time, key point extraction is performed based on a lower density (second density) to minimize the amount of point cloud data.

[0070] Finally, the extracted feature points and key points are fused, and duplicate data is deleted to achieve point cloud compression. This process not only retains important geometric information in the point cloud, but also reduces the amount of data, providing a more efficient and streamlined data foundation for subsequent analysis and processing, and improving the efficiency of subsequent processing.

[0071] In summary, the three-dimensional space registration method based on point cloud and camera image provided by the present invention has the following technical effects: By constructing a virtual three-dimensional space, importing the camera image file as a two-dimensional image, and initializing the virtual camera component at the same time; obtaining the external parameter data set of the physical camera based on the geodetic coordinate system, and performing view calibration on the virtual camera component according to the external parameter data set; interactively view-calibrated virtual camera components, extracting the parameter information of the virtual camera, and generating point cloud configuration parameter information, including the external parameter matrix and the internal parameter matrix; importing the laser point cloud file as three-dimensional point cloud data, using the point cloud configuration parameter information and the geodetic coordinate system to perform view calibration on the point cloud, and combining the calibrated virtual camera component to establish a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image; based on the preliminary mapping relationship, projecting the three-dimensional point cloud onto the two-dimensional image plane, the interactive user interface synchronizes the feature marker point set, and performs adaptive optimization of the preliminary mapping relationship according to the mapping result, and outputs the final spatial alignment result, thereby achieving the technical effect of improving alignment accuracy and efficiency and avoiding human errors.

[0072] Example 2 Figure 2 This is a schematic diagram of the structure of the three-dimensional space registration device based on point cloud and camera image of the present invention. For example, Figure 1 The flow chart of the three-dimensional space registration method based on point cloud and camera image in the present invention can be shown as follows: Figure 2 The structure shown is implemented.

[0073] Based on the same concept as the three-dimensional space registration method based on point cloud and camera image in the above embodiment, the present invention also provides a three-dimensional space registration device based on point cloud and camera image, which includes: The space establishment module 11 is used to establish a virtual three-dimensional space, import camera image files as two-dimensional images, and initialize virtual camera components.

[0074] The camera view calibration module 12 is configured to obtain an extrinsic parameter data set of a physical camera based on a geodetic coordinate system, and perform view calibration of the virtual camera assembly according to the extrinsic parameter data set.

[0075] The configuration parameter extraction module 13 is used to extract virtual camera parameter information from the interactively calibrated virtual camera component and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an external parameter matrix and an internal parameter matrix.

[0076] The calibration and preliminary mapping module 14 is used to import the laser point cloud file into a three-dimensional point cloud, perform point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establish a preliminary mapping relationship between the three-dimensional point cloud and the image points in the two-dimensional image in combination with the calibrated virtual camera component.

[0077] The adaptive optimization module 15 is used to map the three-dimensional point cloud to the two-dimensional image plane according to the preliminary mapping relationship, synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output the spatial registration result.

[0078] In some embodiments, the camera view calibration module 12 includes: The external parameter data set determination unit is used to interact with the camera control log of the physical camera, extract the control G-code of the camera pan / tilt, and determine the external parameter data set.

[0079] A virtual camera view adjustment unit is used to adjust the direction of the virtual camera component according to the external reference data set, and adjust the viewing angle, position and focal length parameters of the virtual camera component to make the virtual camera viewing angle consistent with the viewing angle of the two-dimensional image.

[0080] In some embodiments, the calibration and preliminary mapping module 14 includes: The three-dimensional point cloud importing unit is used to import the laser point cloud file into the three-dimensional point cloud in the virtual three-dimensional space according to the coordinate information of the laser point cloud file.

[0081] A preliminary mapping relationship generating unit is used to define the translation, rotation and affine transformation logic of the three-dimensional point cloud and the two-dimensional image based on the point cloud configuration parameter information, and output the preliminary mapping relationship.

[0082] In some embodiments, the adaptive optimization module 15 includes: The point cloud projection image acquisition unit is used to project the points in the three-dimensional point cloud onto the two-dimensional image plane according to the preliminary mapping relationship to acquire a point cloud projection image.

[0083] A point association relationship establishing unit is used to determine a reference point set in the two-dimensional image based on the feature mark point set, determine a comparison point set in the point cloud projection image, and establish a point association relationship between the reference point set and the comparison point set.

[0084] The preliminary mapping relationship optimization unit is used to combine the optimization algorithm, take the optimization of the multi-scale geometric features of the corresponding points in the reference point set and the comparison point set as the goal, take the change of the reference point set as the penalty factor, and iteratively optimize the preliminary mapping relationship.

[0085] The two-dimensional tuning parameter calculation unit is used to calculate the position difference of the feature marker point set before and after optimization based on the optimal optimization result when the preliminary mapping relationship after iterative optimization meets the preset multi-scale geometric feature target or the preset number of iterative optimizations, and output it as a two-dimensional tuning parameter.

[0086] The spatial registration result output unit is used to perform inverse mapping according to the two-dimensional tuning parameters and output the spatial registration result.

[0087] In some embodiments, the device also includes a downsampling unit, which is used to: interactively use a user interface to obtain point cloud sampling constraints, wherein the point cloud sampling constraints include a first downsampling parameter for key points and a second downsampling parameter for non-key points; interactively use a user interface to obtain point cloud annotation information annotated by professional and technical personnel, wherein the point cloud annotation information includes focus object annotation and controlled object annotation; based on the first downsampling parameter and the second downsampling parameter, downsampling processing is performed on the point cloud areas corresponding to the focus object annotation and the controlled object annotation, respectively.

[0088] In some implementations, the device further includes a slicing and secondary indexing unit, configured to: identify a region of interest based on the two-dimensional image, and adaptively perform grid division based on the identification result to obtain a two-dimensional divided grid and a grid image; adaptively slice and optimize the three-dimensional point cloud based on the two-dimensional divided grid and the spatial registration result; store structured data of the sliced ​​three-dimensional point cloud and the grid image, and establish an index file; receive a ranging request from a user interface, extract the pixel coordinate positions of the measuring points, and perform grid matching based on the pixel coordinate positions of the measuring points and the index file to locate a target slicing area; correspondingly call the stored structured data as target point cloud data, respond to the ranging request based on the target point cloud data and the corresponding grid image, and obtain a ranging result.

[0089] Furthermore, the slicing and secondary indexing unit is also used to: traverse the three-dimensional point cloud after slicing, calculate the shape feature parameters of each point according to the preset k-neighborhood parameters; judge the shape feature parameters according to the preset local shape maximum allowable value; if the shape feature parameters are greater than or equal to the local shape maximum allowable value, extract the feature points of the corresponding area based on the first density; if the shape feature parameters are less than the local shape maximum allowable value, extract the key points of the corresponding area based on the second density; fuse the feature points and the key points, and delete duplicate data to achieve point cloud compression.

[0090] It should be understood that the embodiments mentioned in this specification focus on their differences from other embodiments. The specific embodiments in the aforementioned embodiment one are also applicable to the three-dimensional space registration device based on point cloud and camera image described in embodiment two. For the sake of brevity of the specification, no further elaboration is given here.

[0091] It should be understood that the embodiments disclosed in the present invention and the above description can enable those skilled in the art to use the present invention to implement the present invention. At the same time, the present invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention and are all included in the scope of protection of the present invention.

Claims

1. A three-dimensional space registration method based on point cloud and camera image, characterized in that: The method comprises: Create a virtual 3D space, import the camera image file as a 2D image, and initialize the virtual camera component; Acquire an external parameter data set of a physical camera based on a geodetic coordinate system, and perform view calibration of the virtual camera assembly according to the external parameter data set; After interactive calibration, the virtual camera component extracts virtual camera parameter information and outputs it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an external parameter matrix and an internal parameter matrix; Importing a laser point cloud file as a three-dimensional point cloud, performing point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and image points in the two-dimensional image in combination with the calibrated virtual camera component; The three-dimensional point cloud is mapped to a two-dimensional image plane according to the preliminary mapping relationship, and a feature marker point set is synchronized through an interactive user interface. The preliminary mapping relationship is adaptively optimized in combination with the mapping result, and a spatial registration result is output.

2. The three-dimensional space registration method based on point cloud and camera image according to claim 1, characterized in that: Acquiring an external parameter data set of a physical camera based on a geodetic coordinate system, and performing view calibration of the virtual camera assembly according to the external parameter data set, including: The camera control log of the interactive physical camera is used to extract the control G-code of the camera gimbal and determine the external reference data set; The virtual camera component is pointed according to the external reference data set, and the viewing angle, position and focal length parameters of the virtual camera component are adjusted so that the viewing angle of the virtual camera is consistent with the viewing angle of the two-dimensional image.

3. The three-dimensional space registration method based on point cloud and camera image according to claim 2, characterized in that: Importing a laser point cloud file as a three-dimensional point cloud, performing point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and image points in the two-dimensional image in combination with the calibrated virtual camera component, including: Importing the laser point cloud file into a three-dimensional point cloud in the virtual three-dimensional space according to coordinate information of the laser point cloud file; Based on the point cloud configuration parameter information, the translation, rotation and affine transformation logic of the three-dimensional point cloud and the two-dimensional image are defined, and the output is the preliminary mapping relationship.

4. The three-dimensional space registration method based on point cloud and camera image according to claim 3, characterized in that: Mapping the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, synchronizing a feature marker point set through an interactive user interface, performing adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and outputting a spatial registration result, including: Projecting the points in the three-dimensional point cloud onto the two-dimensional image plane according to the preliminary mapping relationship to obtain a point cloud projection image; Based on the feature mark point set, a reference point set is determined in the two-dimensional image, a comparison point set is determined in the point cloud projection image, and a point position association relationship between the reference point set and the comparison point set is established; In combination with an optimization algorithm, the preliminary mapping relationship is iteratively optimized with the goal of optimizing the multi-scale geometric features of corresponding points in the reference point set and the comparison point set, and with the change of the reference point set as a penalty factor; When the preliminary mapping relationship after iterative optimization satisfies the preset multi-scale geometric feature target or the preset number of iterative optimizations, the position difference of the feature marker point set before and after optimization is calculated based on the optimal optimization result and output as a two-dimensional tuning parameter; Inverse mapping is performed according to the two-dimensional tuning parameters, and a spatial registration result is output.

5. The three-dimensional space registration method based on point cloud and camera image according to claim 1, characterized in that: Mapping the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, synchronizing a feature marker point set through an interactive user interface, performing adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and outputting a spatial registration result, before, including: An interactive user interface is provided for obtaining a point cloud sampling constraint, wherein the point cloud sampling constraint includes a first downsampling parameter for a key point and a second downsampling parameter for a non-key point; An interactive user interface is used to obtain point cloud annotation information annotated by professional technicians, wherein the point cloud annotation information includes annotations of objects of interest and annotations of controlled objects; Based on the first downsampling parameter and the second downsampling parameter, downsampling processing is performed on the point cloud areas corresponding to the focus object label and the controlled object label respectively.

6. The three-dimensional space registration method based on point cloud and camera image according to claim 1, characterized in that: The method further comprises: Identify the region of interest based on the two-dimensional image, and adaptively perform grid division based on the identification result to obtain a two-dimensional divided grid and a grid image; Based on the two-dimensional division grid and the spatial registration result, performing adaptive slicing optimization on the three-dimensional point cloud; Storing the sliced ​​three-dimensional point cloud and the grid image in structured data and creating an index file; Receive a distance measurement request from the user interface, extract the pixel coordinate position of the measurement point, and perform grid matching with the index file according to the pixel coordinate position of the measurement point to locate the target slice area; The corresponding stored structured data is the target point cloud data, and the target point cloud data is combined with the corresponding grid image to perform a ranging request response to obtain a ranging result.

7. The three-dimensional space registration method based on point cloud and camera image according to claim 6, characterized in that: Based on the two-dimensional grid division and the spatial registration result, the three-dimensional point cloud is adaptively sliced ​​and optimized, further comprising: Traversing the sliced ​​three-dimensional point cloud, and calculating the shape feature parameters of each point according to the preset k-neighborhood parameters; Determining the shape characteristic parameters according to a preset maximum allowable value of the local shape; If the shape feature parameter is greater than or equal to the maximum allowable value of the local shape, extracting feature points of the corresponding area based on the first density; If the shape feature parameter is less than the maximum allowable value of the local shape, extracting key points of the corresponding area based on the second density; The feature points and the key points are fused and duplicate data is deleted to achieve point cloud compression.

8. A three-dimensional space registration device based on point cloud and camera image, characterized in that: The device is used to execute the three-dimensional space registration method based on point cloud and camera image according to any one of claims 1 to 7, and the device includes: The space establishment module is used to establish a virtual three-dimensional space, import camera image files as two-dimensional images, and initialize the virtual camera component; A camera view calibration module is used to obtain an external parameter data set of a physical camera based on a geodetic coordinate system, and perform view calibration of the virtual camera assembly according to the external parameter data set; A configuration parameter extraction module is used to extract virtual camera parameter information from the interactively calibrated virtual camera assembly and output it as point cloud configuration parameter information, wherein the point cloud configuration parameter information includes an extrinsic parameter matrix and an intrinsic parameter matrix; a calibration and preliminary mapping module for importing a laser point cloud file as a three-dimensional point cloud, performing point cloud view calibration based on the point cloud configuration parameter information and the geodetic coordinate system, and establishing a preliminary mapping relationship between the three-dimensional point cloud and image points in the two-dimensional image in combination with the calibrated virtual camera assembly; An adaptive optimization module is used to map the three-dimensional point cloud to a two-dimensional image plane according to the preliminary mapping relationship, synchronize the feature marker point set with the interactive user interface, perform adaptive optimization of the preliminary mapping relationship in combination with the mapping result, and output the spatial alignment result.

Images