Three-dimensional image reconstruction method and apparatus

Abstract

The disclosure provides a three-dimensional image reconstruction method and device, which belongs to the technical field of image processing. The method comprises the following steps: obtaining first point cloud data and second point cloud data; the sources of the first point cloud data and the second point cloud data are different; determining a target registration requirement based on the first point cloud data and the second point cloud data; determining a target registration strategy from a plurality of point cloud registration strategies based on the target registration requirement, wherein the point cloud registration accuracies of the plurality of point cloud registration strategies are different; registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data; and rendering the point cloud registration data to obtain a three-dimensional image model. The three-dimensional image reconstruction method and device provided by the disclosure can improve the efficiency and accuracy of three-dimensional image reconstruction.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, and more specifically, relates to a three-dimensional image reconstruction method and apparatus. Background Technology

[0002] 3D reconstruction refers to the creation of mathematical models of 3D objects suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the properties of objects in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. This technology can acquire 3D models with color and texture, viewable from any perspective, providing crucial technical support for numerous fields. In the medical field, 3D models allow for more intuitive and comprehensive diagnosis of physical conditions; in the field of history and culture, artifacts can be reconstructed in 3D for scientific research and tourist visits; furthermore, 3D reconstruction technology has broad application prospects in game development, industrial design, aerospace, and marine engineering. Traditional 3D image reconstruction methods have many shortcomings, such as low computational efficiency in complex scenes.

[0003] Therefore, a highly efficient and accurate 3D image reconstruction method is needed. Summary of the Invention

[0004] The purpose of this disclosure is to provide a three-dimensional image reconstruction method and apparatus to improve the efficiency and accuracy of three-dimensional image reconstruction.

[0005] A first aspect of this disclosure provides a three-dimensional image reconstruction method, comprising:

[0006] Acquire first and second point cloud data; the first and second point cloud data have different sources.

[0007] The target registration requirements are determined based on the first and second point cloud data.

[0008] Based on the target registration requirements, a target registration strategy is determined from multiple point cloud registration strategies, where the point cloud registration accuracy of the multiple point cloud registration strategies is different;

[0009] The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data;

[0010] The point cloud registration data is rendered to obtain a 3D image model.

[0011] A second aspect of this disclosure provides a three-dimensional image reconstruction apparatus, comprising:

[0012] The point cloud data acquisition module is used to acquire first point cloud data and second point cloud data; the first point cloud data and second point cloud data have different sources.

[0013] The requirement determination module is used to determine the target registration requirements based on the first point cloud data and the second point cloud data.

[0014] The strategy determination module is used to determine the target registration strategy from multiple point cloud registration strategies based on the target registration requirements, wherein the point cloud registration accuracies of the multiple point cloud registration strategies are different.

[0015] The registration module is used to register the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data.

[0016] The rendering module is used to render the point cloud registration data to obtain a 3D image model.

[0017] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the three-dimensional image reconstruction method described above.

[0018] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the three-dimensional image reconstruction method described above.

[0019] The beneficial effects of the three-dimensional image reconstruction method and apparatus provided in this disclosure are as follows:

[0020] This disclosure utilizes first point cloud data (derived from high-resolution camera images) and second point cloud data (directly generated by a laser sensor) from different sources. By combining their overlap and accuracy requirements, it determines the target registration needs and strategies, adapting to diverse data situations and improving registration flexibility and accuracy. Secondly, it employs a NeRF model as the core for rendering processing, integrating point cloud registration data with information from the first acquired image. By fully utilizing the image's color, density, and depth information, it can generate a visually realistic and detailed 3D image model. Furthermore, this disclosure helps improve overall reconstruction efficiency and quality, providing strong support for the digitization process in numerous fields such as cultural relic preservation and industrial design, and promoting the expansion of data visualization and analysis applications. Therefore, this disclosure can improve the efficiency and accuracy of 3D image reconstruction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic flowchart of a three-dimensional image reconstruction method provided in an embodiment of this disclosure;

[0023] Figure 2 This is a structural block diagram of a three-dimensional image reconstruction apparatus provided in an embodiment of the present disclosure;

[0024] Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0026] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0027] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a three-dimensional image reconstruction method according to an embodiment of the present disclosure. The method includes:

[0028] S101: Obtain the first point cloud data and the second point cloud data; wherein the first point cloud data and the second point cloud data have different sources.

[0029] In this embodiment, the different sources of the first point cloud data and the second point cloud data refer to different acquisition methods and different acquisition devices.

[0030] Acquire the first point cloud data, including:

[0031] In response to receiving the first acquired image sent by the first device;

[0032] The first point cloud data is determined based on the first acquired image.

[0033] The first device can be a camera with a high-resolution lens, and the first acquired image can be a two-dimensional color image. Then, the two-dimensional color image is converted into point cloud data, i.e., the first point cloud data. The conversion of the two-dimensional color image into point cloud data (i.e., the first point cloud data) can be achieved using depth map methods or model-based methods.

[0034] If using the depth map method, the color image can first be converted to grayscale to obtain a grayscale image (i.e., a depth map). A depth map is a grayscale image where each pixel value represents the distance of that point from the camera. By using the coordinates of the pixels in the depth map and their corresponding depth values, the coordinates of each pixel in three-dimensional space can be calculated, thereby generating three-dimensional point cloud data (i.e., the first point cloud data).

[0035] If a two-dimensional image of an object and its corresponding three-dimensional model are known, a model-based approach can be used. This involves matching features in the two-dimensional image with the three-dimensional model, projecting points from the three-dimensional model onto the two-dimensional image, or mapping points from the two-dimensional image onto the surface of the three-dimensional model, thereby generating three-dimensional point cloud data corresponding to the two-dimensional image.

[0036] Acquire second point cloud data, including:

[0037] In response to receiving point cloud data sent by a second device. The second device can be a laser sensor, which can directly obtain laser point cloud data (i.e., the second point cloud data).

[0038] Digital preservation scenarios for cultural relics:

[0039] Example of first point cloud data acquisition: When digitally archiving a precious ancient sculpture, a professional SLR camera equipped with a high-resolution lens was used to take pictures of the sculpture from multiple angles, obtaining a series of two-dimensional color images (i.e., the first acquired images). Subsequently, staff used 3D reconstruction software and a multi-view geometric algorithm based on feature matching to import these two-dimensional color images into the software. The software automatically identified feature points of the sculpture in different images (such as points corresponding to contour edges, texture details, etc.), and calculated the coordinates of these feature points in three-dimensional space based on their geometric relationships. Finally, the first point cloud data representing the surface shape of the sculpture was generated, which can be used as a reference for subsequent more comprehensive recording of the sculpture's appearance and for restoration and other related work.

[0040] Example of acquiring second point cloud data: To obtain more precise size and shape information of the sculpture, a laser scanner (i.e., the second device) is used to scan the sculpture. The laser scanner emits a laser beam onto the surface of the sculpture, and the detector receives the reflected laser light. Based on parameters such as the laser propagation time and scanning angle, laser point cloud data (i.e., second point cloud data) of the sculpture is quickly and directly generated. This data plays an important role in accurately measuring the spatial position of various parts of the sculpture and judging the surface flatness. Furthermore, combined with the first point cloud data generated from camera images, it enables more complete and accurate digital preservation of the sculpture.

[0041] S102: Determine the target registration requirements based on the first point cloud data and the second point cloud data.

[0042] In this embodiment, considering that the first point cloud data and the second point cloud data are essentially heterogeneous data, when registering the two point cloud data in the future, it is necessary to first consider the specific application scenario, data characteristics and other factors, as well as the requirements for registration speed or registration accuracy.

[0043] First, determine the overlap between the first point cloud data and the second point cloud data;

[0044] Then, in response to the registration results of the first point cloud data and the second point cloud data, the first precision must be met:

[0045] If the overlap is greater than or equal to the first value, then the first requirement will be taken as the target registration requirement.

[0046] If the overlap is less than the first value, then the second requirement will be used as the target registration requirement.

[0047] Finally, in response to the requirement that the registration results of the first point cloud data and the second point cloud data need to meet the second accuracy, the third requirement is taken as the target registration requirement.

[0048] Among them, the first precision is less than the second precision, the applicability of the first requirement is less than the applicability of the second requirement, and the applicability of the second requirement is less than the applicability of the third requirement.

[0049] Therefore, before registering the first point cloud data and the second point cloud data, it is necessary to conduct a comprehensive analysis of the first point cloud data and the second point cloud data, taking into account their own data characteristics, differences, and the requirements of the application scenario for the registration results, so as to clarify the target registration requirements.

[0050] S103: Determine the target registration strategy from multiple point cloud registration strategies based on the target registration requirements, wherein the point cloud registration accuracy of the multiple point cloud registration strategies is different.

[0051] In this embodiment, the target registration requirement determined according to step S102 includes a first requirement, a second requirement, or a third requirement.

[0052] The first requirement corresponds to the first point cloud registration strategy, the second requirement corresponds to the second point cloud registration strategy, and the third requirement corresponds to the third point cloud registration strategy.

[0053] Different point cloud registration strategies have different processing methods and implementation principles, and therefore their accuracy will also vary.

[0054] When performing point cloud registration, the target registration requirements based on the current point cloud data and application scenario are clearly defined. Several different point cloud registration strategies are available, each with varying levels of achievable registration accuracy. Therefore, it is necessary to analyze and weigh these different point cloud registration strategies, considering the pre-defined target registration requirements, to select the most suitable strategy for the current situation. This strategy will then be used to achieve accurate registration of the point cloud data.

[0055] S104: Register the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data.

[0056] In this embodiment, the target registration strategy is determined to be a first point cloud registration strategy, a second point cloud registration strategy, or a third point cloud registration strategy according to step S103. That is, the first point cloud data and the second point cloud data can be processed based on the target registration strategy. For example, the correspondence between the first point cloud data and the second point cloud data can be found. By adjusting the position and angle of the first point cloud data and the second point cloud data, the point cloud data that were originally in different coordinate systems can be aligned as accurately as possible in space to achieve a relatively consistent spatial representation state, and finally the point cloud registration data is obtained.

[0057] S105: Render the point cloud registration data to obtain a 3D image model.

[0058] In this embodiment, the point cloud registration data is rendered, including:

[0059] Determine the first depth information based on point cloud registration data;

[0060] Determine the second depth information, density information, and color information based on the first acquired image;

[0061] The second depth information is supplementary to the first depth information, and the first acquired image is a two-dimensional color image.

[0062] The first depth information, the second depth information, the density information, and the color information are input into the first model to obtain a three-dimensional image model; wherein, the first model can be a Neural Radiance Fields (NeRF) model.

[0063] The first depth information is determined based on point cloud registration data, including:

[0064] The point cloud registration data is input into the first neural network model to obtain the first depth information;

[0065] The first neural network model can be a 3D convolutional neural network model (3DCNN).

[0066] Based on the first acquired image, second depth information, density information, and color information are determined, including:

[0067] The first acquired image is input into the second neural network model to obtain the second depth information, density information and color information;

[0068] The second neural network model can be a 2D convolutional neural network (2DCNN).

[0069] Specifically, the point cloud registration data is rendered to obtain a 3D image model. The steps are as follows:

[0070] First, based on a first neural network model (such as 3DCNN) and point cloud registration data, first depth information (such as view density depth estimation) can be obtained, which is the primary depth information. Then, based on a second neural network model, the first acquired image is processed to generate planar segment features (i.e., second depth information, density information, and color information), where the second depth information is supplementary depth information.

[0071] Secondly, the first and second depth information are transformed into feature vectors and explicit space, where the set of points occupied by the feature vectors and explicit space constitutes the initial neural point cloud (each point has a spatial location, a confidence level, and back-projected image features).

[0072] Then, by inputting the initial neural point cloud (constructed from the first depth information and the second depth information), density information and color information into the first model (such as the NeRF model), a three-dimensional image model can be obtained.

[0073] Finally, by constructing and utilizing explicit spatial information, the NeRF model can be significantly improved in terms of both the speed of adapting to 3D point cloud construction and the quality of generated visual information, resulting in fast, accurate, and high-quality 3D models.

[0074] As can be seen from the above, this disclosure, by acquiring first point cloud data (derived from high-resolution camera images) and second point cloud data (directly generated by a laser sensor) from different sources, and combining their overlap and accuracy requirements to determine the target registration needs and strategies, can adapt to diverse data situations and improve registration flexibility and accuracy. Secondly, by using the NeRF model as the core for rendering processing, and integrating the point cloud registration data with the first acquired image information, fully utilizing the image's color, density, and depth information, a visually realistic and detailed 3D image model can be generated. Simultaneously, this disclosure helps improve overall reconstruction efficiency and quality, providing strong support for the digitization process in many fields such as cultural relic protection and industrial design, and promoting the expansion of data visualization and analysis applications. Therefore, this disclosure can improve the efficiency and accuracy of 3D image reconstruction.

[0075] In one embodiment of this disclosure, acquiring first point cloud data includes:

[0076] In response to receiving the first acquired image sent by the first device;

[0077] The first point cloud data is determined based on the first acquired image.

[0078] In one embodiment of this disclosure, determining first point cloud data based on a first acquired image includes:

[0079] Multiple feature values ​​are determined based on the first acquired image;

[0080] If the first feature value among multiple feature values ​​is less than the first preset value, then the first acquired image is subjected to the first image processing to obtain the first image;

[0081] If the first feature value among multiple feature values ​​is greater than or equal to the first preset value, and the first feature value is less than the second preset value, then the first acquired image is used as the second image;

[0082] If the first feature value among multiple feature values ​​is greater than or equal to the second preset value, then the first acquired image is subjected to the second image processing to obtain the third image;

[0083] Determine the first point cloud data based on the first image, the second image, or the third image;

[0084] The first image processing and the second image processing have different processing methods.

[0085] In this embodiment, the first device is designated as a camera with a high-resolution lens, which is used to acquire image data as the original source for generating the first point cloud data.

[0086] The first acquired image is a two-dimensional color image captured by the first device (camera), which contains two-dimensional visual information of the object or scene, such as pixel color, brightness, contrast and other information in the image, which reflect the appearance characteristics of the photographed object.

[0087] Eigenvalues ​​are quantitative indicators used to describe certain characteristics of an image. The first eigenvalue can be brightness, sharpness, or contrast, etc. These eigenvalues ​​can reflect the quality and characteristics of the image from different aspects, thereby affecting the subsequent image processing methods and the generation of the final first point cloud data.

[0088] The first preset value and the second preset value are pre-set thresholds used to determine the range of the first feature value. They can be dynamically adjusted according to the actual situation. By comparing the first feature value with these two thresholds, the processing method for the first acquired image can be determined.

[0089] The first image processing and the second image processing are different operations performed on the first acquired image according to different situations. The purpose is to optimize the image to better determine the first point cloud data, and the two processing methods differ, such as different filtering operations and enhancement algorithms. Specifically, the first image processing can be histogram equalization or brightness enhancement; the second image processing can be contrast adjustment and the reverse operation of histogram equalization.

[0090] The first image, the second image, and the third image are the result images obtained after the first acquired image has undergone different processing paths. Finally, the first point cloud data is determined based on one of these images.

[0091] Upon receiving the first acquired image from the first device (camera), the process begins to determine the first point cloud data based on this image. First, multiple feature values ​​are calculated from the first acquired image. The key feature value (such as brightness, sharpness, or contrast) is used to determine the image's state. If the first feature value is less than a first preset value, it indicates that the image performs poorly in that feature (e.g., too dark), and the first acquired image undergoes first image processing to obtain the first image. If the first feature value is between the first and second preset values, it means the image is in a suitable brightness state for that feature, and the first acquired image is directly used as the second image. If the first feature value is greater than or equal to the second preset value, it indicates that the image performs well in that feature (e.g., sufficient brightness), and the first acquired image undergoes second image processing to obtain the third image. Finally, the first point cloud data is determined based on the obtained first, second, or third image. Different point cloud generation algorithms or parameter settings can be used for different images to adapt to their different characteristics.

[0092] For example, suppose the first device is a high-resolution camera used to photograph cultural relics, and the first captured image is a photograph of an ancient ceramic artifact. The first preset value is set to 30 (brightness range 0-100), and the second preset value is 70.

[0093] The brightness feature value of the first acquired image is calculated to be 20, which is less than a first preset value. At this point, first image processing is performed, such as using a brightness enhancement algorithm to increase the overall brightness of the image, resulting in the first image. Then, using an image-based point cloud generation algorithm (such as multi-view geometry), based on the first image, the coordinates of these feature points in three-dimensional space are calculated by identifying feature points of the ceramic artifact from different perspectives in the image and combining them with camera parameters. This determines the first point cloud data, which can well reflect the shape of the artifact and can be used for the subsequent three-dimensional digital preservation and display of the artifact.

[0094] As can be seen from the above, this disclosure differentiates and processes images based on their feature values, enabling targeted image optimization. For images with low feature values, the first image processing improves the difficulty of point cloud generation caused by insufficient features, thus enhancing point cloud quality. When the feature values ​​are moderate, the original image is directly used as the second image, reducing unnecessary processing and improving efficiency. When the feature values ​​are high, the second image processing facilitates the extraction of more accurate point clouds. Furthermore, different processing methods flexibly adapt to diverse image conditions; whether the image quality is poor or high, it can be effectively converted into first point cloud data, enhancing the system's compatibility and adaptability to image data from different sources.

[0095] In one embodiment of this disclosure, determining a target registration strategy from multiple point cloud registration strategies based on target registration requirements includes:

[0096] Determine the overlap between the first point cloud data and the second point cloud data;

[0097] If the overlap is greater than or equal to the third preset value, then the first point cloud registration strategy will be used as the target registration strategy.

[0098] If the overlap is less than the third preset value, then the second point cloud registration strategy will be used as the target registration strategy.

[0099] Among them, the accuracy of the first point cloud registration strategy is less than that of the second point cloud registration strategy.

[0100] In this embodiment, the overlap refers to the proportion or degree to which the first point cloud data and the second point cloud data cover and overlap each other in space. It reflects the similarity and correlation between the two point cloud data. The higher the overlap, the more common parts the two point cloud data have in space, which facilitates registration.

[0101] The third preset value is a pre-set threshold that can be dynamically adjusted according to the actual situation. It is used to determine the degree of overlap between the first and second point cloud data, thereby deciding which point cloud registration strategy to choose. Although the accuracy of the first point cloud registration strategy is lower than that of the second point cloud registration strategy, the processing speed of the first point cloud registration strategy is faster than that of the second point cloud registration strategy.

[0102] First, the overlap between the first and second point cloud data needs to be calculated. The target registration strategy is determined by comparing the overlap with a third preset value. If the overlap is greater than or equal to the third preset value, it indicates that the two point cloud data have a large overlap. In this case, the first point cloud registration strategy is selected as the target registration strategy because the large overlap area allows even the slightly less accurate first point cloud registration strategy to complete the registration work well, and it also has advantages in terms of calculation speed. If the overlap is less than the third preset value, it means that the two point cloud data have less overlap. In this case, a more accurate point cloud registration strategy is needed to ensure the overall registration effect. Therefore, the more accurate second point cloud registration strategy is selected as the target registration strategy.

[0103] As can be seen from the above, when the overlap is high, the first point cloud registration strategy can be used to quickly register by utilizing more overlapping information, improving processing efficiency and saving time costs. Secondly, when the overlap is low, the second point cloud registration strategy is activated to handle complex situations with high precision and ensure registration accuracy. Therefore, this embodiment is highly adaptable and can be flexibly selected according to the actual data conditions to optimize the registration process, providing a reliable and efficient registration solution for 3D point cloud processing in different fields.

[0104] In one embodiment of this disclosure, the target registration strategy is a first point cloud registration strategy;

[0105] Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data, including:

[0106] The first matrix is ​​determined based on the first point cloud data and the second point cloud data;

[0107] The point cloud registration data is determined by calculating the eigenvectors of the first matrix.

[0108] In this embodiment, the point cloud registration data is the fused data obtained after registration processing, which can accurately reflect the correspondence between the first point cloud data and the second point cloud data in the same coordinate system, and can be used for subsequent analysis, processing and application.

[0109] The first matrix is ​​obtained by specific calculations or transformations of the first point cloud data and the second point cloud data. This matrix contains certain correlation information between the two point cloud data and is the basis for subsequent calculations of point cloud registration data.

[0110] The eigenvectors of the first matrix are non-zero vectors that, under the linear transformation corresponding to that matrix, retain their orientation or undergo only a scaling transformation. In point cloud registration, the eigenvectors of the first matrix can be used to extract key features and orientation information from point cloud data to help determine the accurate registration relationship between point clouds.

[0111] In this embodiment, point cloud registration based on principal component analysis (PCA) can be used as the first point cloud registration strategy. PCA is a data analysis technique used here for point cloud registration. Its core idea is to find the principal components (i.e., the directions with the largest variance) in the point cloud registration data by performing eigenvalue decomposition on the covariance matrix of the point cloud registration data. In point cloud registration, the point cloud shape and orientation information reflected by these principal components can be used to determine the registration relationship.

[0112] In the registration process based on principal component analysis, the first step is to combine the first and second point cloud data to construct a covariance matrix (the first matrix). For example, the point coordinate information in the two point cloud datasets is integrated and processed to calculate the covariance matrix, which reflects the overall spatial distribution of the point cloud.

[0113] The constructed covariance matrix is ​​subjected to eigenvalue decomposition to obtain its eigenvectors. Then, based on the main directional information of the point cloud represented by these eigenvectors, the rotation and translation parameters between the two point cloud datasets are calculated. For example, if an eigenvector represents a main axial direction of the point cloud, the difference in that axial direction between the two point cloud datasets is compared to determine the required rotation angle and translation distance. The first point cloud dataset is then transformed to obtain the registered point cloud dataset, which is then registered with the second point cloud dataset.

[0114] As can be seen from the above, this embodiment, by constructing a first matrix and using its eigenvectors to calculate registration, can effectively mine the inherent shape and orientation information of point cloud data, accurately determine the registration relationship, and improve registration accuracy. Secondly, the first point cloud registration strategy can fully leverage its advantages when processing point cloud data with a certain degree of overlap, quickly completing registration and improving efficiency.

[0115] In one embodiment of this disclosure, the target registration strategy is a second point cloud registration strategy;

[0116] Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data, including:

[0117] Determine the overlapping area of ​​the first point cloud data and the second point cloud data;

[0118] Determine matching feature points based on overlapping regions;

[0119] Point cloud registration data is determined based on matching feature points.

[0120] In this embodiment, the overlapping region refers to the part in three-dimensional space where the first point cloud data and the second point cloud data cover and overlap each other. This region is crucial for determining the correspondence between the first point cloud data and the second point cloud data and is the basic range for subsequent searching of matching feature points.

[0121] Within the overlapping region, points that correspond to each other in the two point cloud datasets are found through specific algorithms or feature extraction methods. These points usually have obvious geometric features or uniqueness in local regions, such as corner points or edge points. Their matching relationship helps to determine the relative position and orientation transformation of the two point cloud datasets.

[0122] This embodiment can employ model-based geometric feature invariant point cloud registration as the second point cloud registration strategy. Geometric feature invariants refer to certain geometric features (such as the distance ratio between points, angular relationships, etc.) that remain unchanged when an object undergoes transformations such as translation, rotation, and scaling. By extracting and utilizing these invariants, matching relationships can be found between different point cloud data, thereby achieving registration.

[0123] This embodiment compares the coordinate ranges of each point in two point cloud datasets to find the spatially overlapping parts and determine their boundaries and ranges. This is the basic step of registration, and subsequent feature point matching will mainly be performed within the overlapping area.

[0124] In the overlapping region, feature points are extracted based on geometric feature invariants. The correspondence between these feature points in the two point cloud datasets is determined by calculating the geometric relationship invariants between them.

[0125] Then, using the coordinate information of the matching feature points in the two point cloud datasets, the rotation and translation parameters required to transform the first point cloud dataset to align with the second point cloud dataset are calculated. For example, the transformation relationship between the matching feature points is fitted using methods such as least squares, and then the first point cloud dataset is rotated and translated according to the calculated parameters to obtain the point cloud registration data.

[0126] As can be seen from the above, this embodiment can accurately locate the correspondence between the first and second point cloud data by finding matching feature points, greatly improving the registration accuracy. Secondly, this strategy focuses on overlapping areas, reducing unnecessary computation and improving registration efficiency. Furthermore, it is highly adaptable to point cloud data from different sources but with similar geometric structures. Whether it is design models and physical object scanning data, or data collected from different perspectives, it can be effectively registered, enhancing the versatility of point cloud registration in multiple fields and providing a reliable, efficient, and accurate registration method for 3D point cloud processing.

[0127] In one embodiment of this disclosure, the three-dimensional image reconstruction method further includes:

[0128] If the registration results of the first point cloud data and the second point cloud data need to meet the second condition, then the third point cloud registration strategy will be used as the target registration strategy.

[0129] Among them, the accuracy of the third point cloud registration strategy is greater than that of the second point cloud registration strategy.

[0130] In this embodiment, the second condition is a pre-set standard setting for determining whether the registration is up to standard. It can be that the second precision is greater than the first precision. Essentially, it requires the registration to reach a relatively higher level of precision in order to measure whether the current registration result meets expectations and whether it can meet the requirements of subsequent 3D image reconstruction for data accuracy.

[0131] The target registration strategy is the third-point cloud registration strategy;

[0132] Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data, including:

[0133] The first distance is determined based on the first point cloud data and the second point cloud data;

[0134] Iteratively determine the point cloud registration data based on the first distance;

[0135] If the iteration result equals the fourth preset value, the iteration stops;

[0136] If the iteration result is less than the fourth preset value, the iteration continues.

[0137] The third point cloud registration strategy can be the iterative nearest point method. This method aims to find the best spatial transformation relationship between two different point cloud data through continuous iterative optimization, so that they can be aligned as accurately as possible in the same coordinate system. Its core lies in repeatedly calculating and adjusting based on information such as the distance between point clouds.

[0138] The first distance, in the iterative nearest point method, is a metric used to determine the distance between points based on the coordinates of points in the first and second point cloud datasets through a specific calculation method. This distance reflects the degree of difference between the two point cloud datasets in their current state and is an important basis for subsequent iterative adjustments. A common example is the Euclidean distance (the straight-line distance between two points in space).

[0139] Iteration refers to repeatedly executing the same or similar calculation steps according to certain rules and algorithms. Each iteration adjusts the results based on the previous one, with the aim of gradually optimizing the registration effect between point cloud data, making them closer and closer to an accurate alignment. In the iterative nearest-point method, each iteration updates the transformation relationship (such as rotation and translation parameters) between point clouds based on information such as the current distance.

[0140] The fourth preset value is a pre-defined threshold used to determine whether the iteration process has reached the expected goal and whether it can be stopped. It is usually related to the registration accuracy, and for example, it can be a value representing the average distance between point clouds. When the distance-related index calculated during the iteration process reaches this preset value, the registration accuracy is considered to meet the requirements, and the iteration can end.

[0141] Using the calculated first distance as a reference, the iterative calculation begins according to the iterative nearest point method. In each iteration, the system searches for the closest point pair between two point cloud datasets based on the current distance information. Then, based on the relationship between these point pairs, it calculates and updates the rotation and translation parameters that enable better alignment of the two point cloud datasets. This process is repeated continuously to gradually optimize the registration effect until the accurately aligned point cloud registration data is finally determined.

[0142] After each iteration, a result related to the current registration state (such as the average distance between point clouds) is obtained. This result is compared with a pre-set first preset value. If the two are equal, it means that the current registration accuracy has met the expected requirements, and the entire iteration process can be terminated, indicating that point cloud registration data that meets the accuracy expectations has been obtained.

[0143] When the result obtained after each iteration (the relevant index reflecting the current registration degree) is less than the first preset value, it indicates that the current registration effect has not yet reached the preset accuracy standard, and the two point cloud data need to be further adjusted and aligned. Therefore, the next round of iteration calculation should be carried out according to the iteration rules, and this judgment and iteration process should be repeated until the stopping condition is met.

[0144] As can be seen from the above, this embodiment flexibly switches registration strategies based on the registration results. When the accuracy requirement increases, the third point cloud registration strategy is activated, ensuring that the high-precision registration needs are met. Secondly, the third point cloud registration strategy, based on distance iterative optimization, can gradually and accurately adjust the point cloud data, effectively improving registration accuracy and reducing error accumulation. Furthermore, by using preset values ​​to determine the iteration stopping condition, the entire registration process is controllable and efficient, avoiding over-computation. While ensuring accuracy, it also takes into account processing efficiency, enhancing the adaptability and reliability of 3D image reconstruction under different accuracy requirement scenarios.

[0145] Considering that there are both overlapping and non-overlapping areas between the first and second point cloud data, the fourth point cloud registration strategy can be used as the target registration strategy, which is also a comprehensive registration strategy.

[0146] The target registration strategy is the fourth-point cloud registration strategy;

[0147] Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data, including:

[0148] Determine the overlapping area of ​​the first point cloud data and the second point cloud data as the first region;

[0149] Determine the non-overlapping area between the first point cloud data and the second point cloud data, and designate it as the second region;

[0150] The first point cloud registration strategy is used as the first target registration strategy for the first region.

[0151] The third point cloud registration strategy is used as the second target registration strategy for the second region.

[0152] Based on the first point cloud registration strategy, the first point cloud data and the second point cloud data of the first region are processed to obtain the first point cloud registration data.

[0153] The first and second point cloud data of the second region are processed based on the second point cloud registration strategy to obtain the second point cloud registration data.

[0154] The first point cloud registration data and the second point cloud registration data are combined to obtain the target point cloud data.

[0155] In this embodiment, the first point cloud registration strategy (such as point cloud registration based on principal component analysis) can utilize principal component analysis technology to identify the main component directions in the first and second point cloud data of the first region by performing eigenvalue decomposition on the covariance matrix of the data. Then, based on these directional information, the transformation relationship between the point clouds can be determined. When applied to the first region, it can quickly extract the main characteristic directions of the overlapping point clouds, and preliminarily align the two point cloud data based on these directions. Its advantage lies in its relatively high computational efficiency and the ability to quickly obtain a relatively reasonable registration result when data quality requirements are not extremely stringent.

[0156] The third point cloud registration strategy (such as the iterative nearest point method) is a high-precision registration method based on the idea of ​​continuous iterative optimization. When applied to the second region, it repeatedly calculates the nearest point pairs between point clouds and updates the transformation matrix according to the relationship between these point pairs, gradually reducing the difference between the two point cloud data to achieve extremely high registration accuracy.

[0157] Taking point cloud registration in cultural relic restoration as an example:

[0158] By using professional point cloud processing software to analyze the coordinate information of the two point cloud data, it was determined that a certain range around the neck and shoulders of the statue is the overlapping area (first area), while the details of the head of the statue (only present in the second point cloud data) and most of the body (only present in the first point cloud data) are the non-overlapping areas (second area).

[0159] For the first region (overlapping region), a point cloud registration strategy based on principal component analysis is adopted. The covariance matrix of the two point cloud datasets in this region is calculated, and eigenvectors are extracted to determine the main directions, such as the axial direction determined by the statue's body posture. Rotation and translation parameters are then calculated to initially align the two point cloud datasets in this region, resulting in the first registered point cloud dataset. For the second region (non-overlapping region), the iterative nearest point method is used. Initial transformation parameters are estimated for the point cloud data of the statue's face. Then, the nearest point pairs are continuously searched. After multiple iterations, the facial detail point cloud data acquired by the handheld scanner and the body part point cloud data scanned from a distance achieve high-precision registration in the non-overlapping region, resulting in the second registered point cloud dataset.

[0160] The first point cloud registration data and the second point cloud registration data are combined to form complete target point cloud data.

[0161] As can be seen from the above, employing adaptation strategies for overlapping and non-overlapping regions of point cloud data can improve overall efficiency and ensure the accuracy of key parts. Furthermore, by fully utilizing the strengths of different strategies and adapting to complex data conditions, the quality and accuracy of 3D image reconstruction models can be effectively improved.

[0162] Corresponding to the three-dimensional image reconstruction method in the above embodiments, Figure 2 This is a structural block diagram of a three-dimensional image reconstruction apparatus provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 2 The 3D image reconstruction device 20 includes: a point cloud data acquisition module 21, a requirement determination module 22, a strategy determination module 23, a registration module 24, and a rendering module 25.

[0163] The point cloud data acquisition module 21 is used to acquire first point cloud data and second point cloud data; the first point cloud data and second point cloud data have different sources.

[0164] Requirement determination module 22 is used to determine target registration requirements based on the first point cloud data and the second point cloud data;

[0165] The strategy determination module 23 is used to determine the target registration strategy from multiple point cloud registration strategies based on the target registration requirements, wherein the point cloud registration accuracies of the multiple point cloud registration strategies are different.

[0166] Registration module 24 is used to register the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data;

[0167] The rendering module 25 is used to render the point cloud registration data to obtain a 3D image model.

[0168] In one embodiment of this disclosure, the point cloud data acquisition module 21 is specifically used for:

[0169] In response to receiving the first acquired image sent by the first device;

[0170] The first point cloud data is determined based on the first acquired image.

[0171] In one embodiment of this disclosure, the point cloud data acquisition module 21 is further configured to:

[0172] Multiple feature values ​​are determined based on the first acquired image;

[0173] If the first feature value among multiple feature values ​​is less than the first preset value, then the first acquired image is subjected to the first image processing to obtain the first image;

[0174] If the first feature value among multiple feature values ​​is greater than or equal to the first preset value, and the first feature value is less than the second preset value, then the first acquired image is used as the second image;

[0175] If the first feature value among multiple feature values ​​is greater than or equal to the second preset value, then the first acquired image is subjected to the second image processing to obtain the third image;

[0176] Determine the first point cloud data based on the first image, the second image, or the third image;

[0177] The first image processing and the second image processing have different processing methods.

[0178] In one embodiment of this disclosure, the strategy determination module 23 is specifically used for:

[0179] Determine the overlap between the first point cloud data and the second point cloud data;

[0180] If the overlap is greater than or equal to the third preset value, then the first point cloud registration strategy will be used as the target registration strategy.

[0181] If the overlap is less than the third preset value, then the second point cloud registration strategy will be used as the target registration strategy.

[0182] Among them, the accuracy of the first point cloud registration strategy is less than that of the second point cloud registration strategy.

[0183] In one embodiment of this disclosure, the target registration strategy is a first point cloud registration strategy;

[0184] Registration module 24 is specifically used for:

[0185] The first matrix is ​​determined based on the first point cloud data and the second point cloud data;

[0186] The point cloud registration data is determined by calculating the eigenvectors of the first matrix.

[0187] In one embodiment of this disclosure, the target registration strategy is a second point cloud registration strategy;

[0188] Registration module 24 is also used for:

[0189] Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data, including:

[0190] Determine the overlapping area of ​​the first point cloud data and the second point cloud data;

[0191] Determine matching feature points based on overlapping regions;

[0192] Point cloud registration data is determined based on matching feature points.

[0193] In one embodiment of this disclosure, the strategy determination module 23 is further configured to:

[0194] If the registration results of the first point cloud data and the second point cloud data need to meet the second condition, then the third point cloud registration strategy will be used as the target registration strategy.

[0195] Among them, the accuracy of the third point cloud registration strategy is greater than that of the second point cloud registration strategy.

[0196] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of modules 21 to 25 are shown.

[0197] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0198] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0199] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0200] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this disclosure can execute the implementation methods described in the first and second embodiments of the three-dimensional image reconstruction method provided in the embodiments of this disclosure, or they can execute the implementation methods of the electronic devices described in the embodiments of this disclosure, which will not be repeated here.

[0201] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0202] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0203] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0204] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0206] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0207] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0208] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A three-dimensional image reconstruction method, characterized in that, include: In response to receiving the first acquired image sent by the first device; Multiple feature values ​​are determined based on the first acquired image; The first device is a camera, and the first acquired image is a two-dimensional color image; If the first feature value among the plurality of feature values ​​is less than a first preset value, then the first acquired image is subjected to first image processing to obtain a first image; If the first feature value among the plurality of feature values ​​is greater than or equal to the first preset value, and the first feature value is less than the second preset value, then the first acquired image is used as the second image; If the first feature value among the plurality of feature values ​​is greater than or equal to the second preset value, then the first acquired image is subjected to second image processing to obtain a third image; First point cloud data is determined based on the first image, the second image, or the third image; wherein the processing methods for the first image processing and the second image processing are different; In response to receiving second point cloud data sent by a second device, the second device being a laser sensor, the second point cloud data being laser point cloud data; The target registration requirements are determined based on the first point cloud data and the second point cloud data; Based on the target registration requirement, a target registration strategy is determined from multiple point cloud registration strategies, wherein the point cloud registration accuracies of the multiple point cloud registration strategies are different. Based on the target registration strategy, the first point cloud data and the second point cloud data are registered to obtain point cloud registration data; The point cloud registration data is rendered to obtain a three-dimensional image model.

2. The three-dimensional image reconstruction method as described in claim 1, characterized in that, The step of determining the target registration strategy from multiple point cloud registration strategies based on the target registration requirement includes: Determine the overlap between the first point cloud data and the second point cloud data; If the overlap is greater than or equal to the third preset value, then the first point cloud registration strategy is used as the target registration strategy. If the overlap is less than the third preset value, then the second point cloud registration strategy will be used as the target registration strategy. The accuracy of the first point cloud registration strategy is less than that of the second point cloud registration strategy.

3. The three-dimensional image reconstruction method as described in claim 2, characterized in that, The target registration strategy is the first point cloud registration strategy; The process of registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data includes: The first matrix is ​​determined based on the first point cloud data and the second point cloud data; The point cloud registration data is determined by calculating based on the feature vectors of the first matrix.

4. The three-dimensional image reconstruction method as described in claim 2, characterized in that, The target registration strategy is the second point cloud registration strategy; The process of registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data includes: Determine the overlapping area between the first point cloud data and the second point cloud data; Matching feature points are determined based on the overlapping region; The point cloud registration data is determined based on the matching feature points.

5. The three-dimensional image reconstruction method as described in claim 2, characterized in that, Also includes: If the registration results of the first point cloud data and the second point cloud data need to meet the second condition, then the third point cloud registration strategy will be used as the target registration strategy. The accuracy of the third point cloud registration strategy is greater than that of the second point cloud registration strategy.

6. A three-dimensional image reconstruction device, characterized in that, include: The point cloud data acquisition module is used in response to receiving the first acquired image sent by the first device; Multiple feature values ​​are determined based on the first acquired image; If the first feature value among the plurality of feature values ​​is less than a first preset value, then the first acquired image is subjected to first image processing to obtain a first image; If the first feature value among the plurality of feature values ​​is greater than or equal to the first preset value, and the first feature value is less than the second preset value, then the first acquired image is used as the second image; If the first feature value among the plurality of feature values ​​is greater than or equal to the second preset value, then the first acquired image is subjected to second image processing to obtain a third image; First point cloud data is determined based on the first image, the second image, or the third image; wherein the processing methods for the first image processing and the second image processing are different; In response to receiving second point cloud data sent by a second device, the second device being a laser sensor, the second point cloud data being laser point cloud data; The requirement determination module is used to determine the target registration requirement based on the first point cloud data and the second point cloud data. The strategy determination module is used to determine a target registration strategy from multiple point cloud registration strategies based on the target registration requirements, wherein the multiple point cloud registration strategies have different point cloud registration accuracies. The registration module is used to register the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data; The rendering module is used to render the point cloud registration data to obtain a three-dimensional image model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images