Three-dimensional image reconstruction method and device

Abstract

The invention provides a three-dimensional image reconstruction method and device, and belongs to the technical field of image processing, and the method comprises the steps: obtaining first point cloud data and second point cloud data; wherein the sources of the first point cloud data and the second point cloud data are different; determining a target registration demand based on the first point cloud data and the second point cloud data; a target registration strategy is determined from a plurality of point cloud registration strategies based on the target registration requirement, and the point cloud registration precisions of the plurality of point cloud registration strategies are different; registering the first point cloud data and the second point cloud data based on a target registration strategy to obtain point cloud registration data; and performing rendering processing on the point cloud registration data to obtain a three-dimensional image model. According to the three-dimensional image reconstruction method and device provided by the invention, the efficiency and accuracy of three-dimensional image reconstruction can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, to a three-dimensional image reconstruction method and device. Background Art

[0002] Three-dimensional reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects. It is the basis for processing, operating and analyzing its properties in a computer environment, and is also a key technology for establishing virtual reality in computers to express the objective world. This technology can obtain three-dimensional models with color texture observed from any perspective, providing important technical support for many fields. In the medical field, the use of three-dimensional models can diagnose physical conditions more intuitively and comprehensively; in the field of history and culture, cultural relics can be reconstructed in three dimensions for scientific research and tourists to visit; in addition, in the fields of game development, industrial design, aerospace and navigation, three-dimensional reconstruction technology also has broad application prospects. Traditional three-dimensional image reconstruction methods have many shortcomings, such as low computational efficiency in complex scenes.

[0003] Therefore, a high-efficiency and high-accuracy three-dimensional image reconstruction method is needed. Summary of the invention

[0004] The purpose of the present disclosure is to provide a three-dimensional image reconstruction method and device to improve the efficiency and accuracy of three-dimensional image reconstruction.

[0005] A first aspect of the embodiments of the present disclosure provides a three-dimensional image reconstruction method, comprising: Acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources; Determining target registration requirements based on the first point cloud data and the second point cloud data; Determining a target registration strategy from a plurality of point cloud registration strategies based on target registration requirements, wherein the point cloud registration accuracies of the plurality of point cloud registration strategies are different; Registering the first point cloud data and the second point cloud data based on a target registration strategy to obtain point cloud registration data; The point cloud registration data is rendered to obtain a three-dimensional image model.

[0006] A second aspect of the embodiments of the present disclosure provides a three-dimensional image reconstruction device, including: A point cloud data acquisition module, used to acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources; A requirement determination module, used to determine a target registration requirement based on the first point cloud data and the second point cloud data; A strategy determination module is used to determine a target registration strategy from a plurality of point cloud registration strategies based on target registration requirements, wherein the point cloud registration accuracies of the plurality of point cloud registration strategies are different; A registration module, used for registering the first point cloud data and the second point cloud data based on a target registration strategy to obtain point cloud registration data; The rendering module is used to render the point cloud registration data to obtain a three-dimensional image model.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned three-dimensional image reconstruction method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned three-dimensional image reconstruction method are implemented.

[0009] The beneficial effects of the 3D image reconstruction method and device provided by the embodiments of the present disclosure are: The present disclosure obtains first point cloud data (derived from high-resolution camera images) and second point cloud data (generated directly by laser sensors) from different sources, and determines the target registration requirements and strategies in combination with the overlap and accuracy requirements of the two, which can adapt to various data situations and improve the flexibility and accuracy of registration. Secondly, rendering processing is performed with the NeRF model as the core, integrating the point cloud registration data and the first acquired image information, and making full use of the color, density information and depth information of the image to generate a three-dimensional image model with realistic visual effects and rich details. At the same time, the present disclosure helps to improve the overall reconstruction efficiency and quality, provide strong support for the digitalization process in many fields such as cultural relics protection and industrial design, and promote the expansion of data visualization and analysis applications. Therefore, the present disclosure can improve the efficiency and accuracy of three-dimensional image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a three-dimensional image reconstruction method provided by an embodiment of the present disclosure; Figure 2 A structural block diagram of a three-dimensional image reconstruction device provided by an embodiment of the present disclosure; Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1 A schematic diagram of a flow chart of a three-dimensional image reconstruction method provided by an embodiment of the present disclosure, the method comprising: S101: Acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources.

[0015] In this embodiment, the different sources of the first point cloud data and the second point cloud data refer to different collection methods and different collection devices.

[0016] Get the first point cloud data, including: In response to receiving a first captured image sent by a first device; First point cloud data is determined based on the first acquired image.

[0017] The first device may be a camera with a high-resolution lens, and the first captured image may be a two-dimensional color image. Afterwards, the two-dimensional color image is converted into point cloud data, i.e., first point cloud data. The two-dimensional color image is converted into point cloud data (i.e., first point cloud data) by using a depth map method or a model-based method.

[0018] If the depth map method is used, the color image can be first converted to grayscale to obtain a grayscale image (i.e., depth map). A depth map is a grayscale image in which each pixel value represents the distance of the point from the camera. Through the coordinates of the pixels in the depth map and the corresponding depth values, the coordinates of each pixel in three-dimensional space can be calculated, thereby generating three-dimensional point cloud data (i.e., first point cloud data).

[0019] If the two-dimensional image of the object and its corresponding three-dimensional model are known, a model-based approach can be used to generate three-dimensional point cloud data corresponding to the two-dimensional image by matching the features in the two-dimensional image with the three-dimensional model, projecting the points in the three-dimensional model onto the two-dimensional image, or mapping the points in the two-dimensional image to the surface of the three-dimensional model.

[0020] Get the second point cloud data, including: In response to receiving the point cloud data sent by the second device, the second device may be a laser sensor, and the laser sensor may directly obtain the laser point cloud data (ie, the second point cloud data).

[0021] Digital protection scenarios for cultural relics: Example of obtaining the first point cloud data: When digitally archiving a precious ancient sculpture, a professional SLR camera equipped with a high-resolution lens was used to shoot around the sculpture from multiple angles to obtain a series of two-dimensional color images (i.e., the first acquired images). Subsequently, the staff used 3D reconstruction software and a multi-view geometry algorithm based on feature matching to import these two-dimensional color images into the software. The software automatically identified the characteristic points of the sculpture in different images (such as points corresponding to the contour edge, texture details, etc.), and calculated the coordinates of these characteristic points in three-dimensional space based on the geometric relationship between them, and finally generated the first point cloud data representing the surface shape of the sculpture, so as to record the appearance of the sculpture more comprehensively and provide reference for related work such as restoration.

[0022] Example of obtaining the second point cloud data: In order to obtain more accurate size and shape information of the sculpture, a laser scanner (i.e., the second device) is used to scan the sculpture. The laser scanner emits a laser beam to the surface of the sculpture, and the detector receives the reflected laser. Based on the propagation time of the laser, scanning angle and other parameters, the laser point cloud data (i.e., the second point cloud data) of the sculpture is quickly and directly generated. This data plays an important role in accurately measuring the spatial position of each part of the sculpture and judging the surface flatness, and combined with the first point cloud data generated by the camera image, it can achieve more complete and accurate digital protection of the sculpture.

[0023] S102: Determine target registration requirements based on the first point cloud data and the second point cloud data.

[0024] In this embodiment, considering that the first point cloud data and the second point cloud data are essentially heterogeneous data of different dimensions, when subsequently aligning the two point cloud data, it is necessary to first consider factors such as the specific application scenario and data characteristics, and it is also possible to consider the requirements for the alignment speed or the alignment accuracy.

[0025] First, determining the overlap between the first point cloud data and the second point cloud data; Then, in response to the registration result of the first point cloud data and the second point cloud data, a first accuracy needs to be satisfied: If the overlap is greater than or equal to the first value, the first requirement is used as the target registration requirement; If the overlap is less than the first value, the second requirement is used as the target registration requirement; Finally, in response to the registration result of the first point cloud data and the second point cloud data needing to meet the second accuracy, the third requirement is used as the target registration requirement; Among them, the first accuracy is smaller than the second accuracy, the applicability of the first requirement is smaller than the applicability of the second requirement, and the applicability of the second requirement is smaller than the applicability of the third requirement.

[0026] Therefore, before registering the first point cloud data and the second point cloud data, it is necessary to first conduct a comprehensive analysis of the first point cloud data and the second point cloud data, taking into account factors such as their own data characteristics, differences, and the requirements of the application scenario for the registration results, so as to clarify the target registration requirements.

[0027] S103: Determine a target registration strategy from a plurality of point cloud registration strategies based on a target registration requirement, wherein the plurality of point cloud registration strategies have different point cloud registration accuracies.

[0028] In this embodiment, the target registration requirement determined in step S102 includes a first requirement, a second requirement, or a third requirement.

[0029] The first requirement corresponds to the first point cloud registration strategy, the second requirement corresponds to the second point cloud registration strategy, and the third requirement corresponds to the third point cloud registration strategy.

[0030] Different point cloud registration strategies have different processing methods and implementation principles, so the accuracy will also be different.

[0031] When performing point cloud registration, the target registration requirements for current point cloud data and application scenarios have been clarified. At the same time, there are a variety of different point cloud registration strategies to choose from, and these strategies differ in the point cloud registration accuracy that can be achieved. On this basis, it is necessary to analyze and weigh these point cloud registration strategies with different accuracies and characteristics based on the previously determined target registration requirements, and select the strategy that best suits the current situation as the target registration strategy, so that the strategy can be used next to achieve accurate registration of point cloud data.

[0032] S104: Registering the first point cloud data and the second point cloud data based on a target registration strategy to obtain point cloud registration data.

[0033] In this embodiment, according to step S103, the target registration strategy is determined to be the first point cloud registration strategy, the second point cloud registration strategy or the third point cloud registration strategy, that is, the first point cloud data and the second point cloud data can be processed based on the target registration strategy, for example, the correspondence between the first point cloud data and the second point cloud data is found, and the position and angle of the first point cloud data and the second point cloud data are adjusted so that the point cloud data originally in different coordinate systems can be spatially aligned as accurately as possible to achieve a relatively consistent spatial representation state, and finally obtain the point cloud registration data.

[0034] S105: Rendering the point cloud registration data to obtain a three-dimensional image model.

[0035] In this embodiment, rendering processing is performed on the point cloud registration data, including: Determining first depth information based on the point cloud registration data; determining second depth information, density information, and color information based on the first acquired image; The second depth information is supplementary information of the first depth information, and the first acquired image is a two-dimensional color image; The first depth information, the second depth information, the density information and the color information are input into the first model to obtain a three-dimensional image model; wherein the first model may be a Neural Radiance Fields (NeRF) model.

[0036] Determining first depth information based on the point cloud registration data includes: Inputting the point cloud registration data into a first neural network model to obtain first depth information; Among them, the first neural network model can be a 3D convolutional neural network model (3D Convolutional Neural Networks, 3DCNN).

[0037] Determining second depth information, density information, and color information based on the first acquired image includes: Inputting the first acquired image into a second neural network model to obtain second depth information, density information, and color information; Among them, the second neural network model can be a 2D convolutional neural network model (2D Convolutional Neural Networks, 2DCNN).

[0038] Specifically, the point cloud registration data is rendered to obtain a three-dimensional image model, and the steps are as follows: First, based on the first neural network model (such as 3DCNN) and point cloud registration data, the first depth information (such as view density depth estimation) can be obtained, which is the main depth information. Then, the first acquired image is processed based on the second neural network model to generate plane fragment features (i.e., second depth information, density information and color information), where the second depth information is supplementary depth information.

[0039] Secondly, the first depth information and the second depth information are converted into a feature vector and an explicit space, wherein the feature vector and the set of points occupied by the explicit space constitute an initial neural point cloud (each point has a spatial position, a confidence and back-projected image features).

[0040] Then, the initial neural point cloud (constructed by the first depth information and the second depth information), density information and color information are input into the first model (such as the NeRF model), and a three-dimensional image model can be obtained.

[0041] Finally, by constructing and utilizing explicit spatial information, the NeRF model can be significantly improved in terms of adapting the speed of building three-dimensional point clouds and generating visual quality, thus obtaining a fast, accurate, and high-quality three-dimensional model.

[0042] From the above, it can be concluded that the present disclosure obtains the first point cloud data (derived from high-resolution camera images) and the second point cloud data (generated directly by laser sensors) from different sources, and determines the target registration requirements and strategies in combination with the overlap and accuracy requirements of the two, which can adapt to various data situations and improve the flexibility and accuracy of registration. Secondly, rendering processing is performed with the NeRF model as the core, integrating the point cloud registration data and the first acquired image information, and making full use of the color, density information and depth information of the image to generate a three-dimensional image model with realistic visual effects and rich details. At the same time, the present disclosure helps to improve the overall reconstruction efficiency and quality, provide strong support for the digitalization process in many fields such as cultural relics protection and industrial design, and promote the expansion of data visualization and analysis applications. Therefore, the present disclosure can improve the efficiency and accuracy of three-dimensional image reconstruction.

[0043] In one embodiment of the present disclosure, obtaining first point cloud data includes: In response to receiving a first captured image sent by a first device; First point cloud data is determined based on the first acquired image.

[0044] In one embodiment of the present disclosure, determining first point cloud data based on a first acquired image includes: determining a plurality of characteristic values ​​according to the first acquired image; If a first eigenvalue among the multiple eigenvalues ​​is less than a first preset value, performing first image processing on the first collected image to obtain a first image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a first preset value, and the first eigenvalue is less than a second preset value, taking the first collected image as the second image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a second preset value, performing second image processing on the first collected image to obtain a third image; Determine first point cloud data based on the first image, the second image, or the third image; The first image processing and the second image processing are performed in different ways.

[0045] In this embodiment, the first device is designated as a camera with a high-resolution lens, which is used to collect image data as an original source for generating the first point cloud data.

[0046] The first captured image is a two-dimensional color image captured by the first device (camera), and contains two-dimensional visual information of an object or scene. For example, information such as pixel color, brightness, and contrast in the image reflects the appearance characteristics of the photographed object.

[0047] The eigenvalue is used to describe the quantitative index of a certain characteristic of the image. The first eigenvalue can be brightness, clarity or contrast, etc. These eigenvalues ​​can reflect the quality and characteristics of the image from different aspects, and thus affect the subsequent processing of the image and the generation of the final first point cloud data.

[0048] The first preset value and the second preset value are pre-set thresholds for determining the range of the first eigenvalue, and can be dynamically adjusted according to actual conditions. By comparing the first eigenvalue with the two thresholds, it is determined what processing method to take for the first captured image.

[0049] The first image processing and the second image processing are different operation modes for the first acquired image according to different situations, and the purpose is to optimize the image to better determine the first point cloud data, and there are differences in the processing modes of the two, such as different filtering operations, enhancement algorithms, etc. Among them, the first image processing can be histogram equalization or brightness enhancement; the second image processing can be contrast adjustment, the reverse operation of histogram equalization.

[0050] The first image, the second image, and the third image are result images obtained after the first acquisition image has been processed through different paths, and the first point cloud data is finally determined based on one of these images.

[0051] After receiving the first captured image sent by the first device (camera), the first point cloud data is determined based on the image. First, multiple feature values ​​are calculated based on the first captured image, among which the first feature value (such as brightness, clarity or contrast) that is focused on is used to judge the state of the image. If the first feature value is less than the first preset value, it means that the image performs poorly in this feature (for example, the brightness is too dark), and the first captured image is processed by the first image to obtain the first image; if the first feature value is between the first preset value and the second preset value, it means that the image feature is in a state of suitable brightness, and the first captured image is directly used as the second image; if the first feature value is greater than or equal to the second preset value, it means that the image performs well in this feature (such as sufficient brightness), and the first captured image is processed by the second image to obtain the third image. Finally, the first point cloud data is determined based on the obtained first image, second image or third image. Different images can use different point cloud generation algorithms or parameter settings to adapt to different characteristics of the image.

[0052] For example, assuming that the first device is a high-resolution camera for photographing cultural relics, and the first captured image is a photograph of an ancient ceramic cultural relic, the first preset value is set to 30 (brightness range 0-100), and the second preset value is set to 70.

[0053] The brightness characteristic value of the first acquired image is calculated to be 20, which is less than the first preset value. At this time, the first image processing is performed, such as using a brightness enhancement algorithm to increase the overall brightness of the image to obtain the first image. Then, an image-based point cloud generation algorithm (such as a multi-view geometry method) is used as the basis to identify the different perspective feature points of the ceramic cultural relics in the image, and the camera parameters are combined to calculate the coordinates of these feature points in three-dimensional space, thereby determining the first point cloud data, which can better reflect the shape of the cultural relics and can be used for the three-dimensional digital protection and display of the cultural relics in the future.

[0054] From the above, it can be concluded that the present disclosure can optimize images in a targeted manner by distinguishing and processing based on image eigenvalues. Performing the first image processing on images with low eigenvalues ​​can improve the difficulty of point cloud generation caused by insufficient features and improve the quality of point clouds; directly using the original image as the second image when the eigenvalue is moderate can reduce unnecessary processing and improve efficiency; performing the second image processing when the eigenvalue is high is conducive to extracting more accurate point clouds. Secondly, different processing methods can flexibly adapt to various image conditions. Whether it is poor quality or high-quality images, they can be effectively converted into the first point cloud data, which enhances the compatibility and adaptability of the entire system to image data from different sources.

[0055] In one embodiment of the present disclosure, a target registration strategy is determined from a plurality of point cloud registration strategies based on target registration requirements, including: determining a degree of overlap between the first point cloud data and the second point cloud data; If the overlap is greater than or equal to the third preset value, the first point cloud registration strategy is used as the target registration strategy; If the overlap is less than the third preset value, the second point cloud registration strategy is used as the target registration strategy; Among them, the accuracy of the first point cloud registration strategy is less than the accuracy of the second point cloud registration strategy.

[0056] In this embodiment, the overlap refers to the proportion or degree of the overlapping parts of the first point cloud data and the second point cloud data in space. It reflects the similarity and correlation between the two point cloud data. The higher the overlap, the more common parts the two point cloud data have in space, which facilitates alignment.

[0057] The third preset value is a pre-set threshold value, which can be dynamically adjusted according to the actual situation, and is used to determine the degree of overlap between the first point cloud data and the second point cloud data, so as to determine which point cloud registration strategy to select. Although the accuracy of the first point cloud registration strategy is lower than that of the second point cloud registration strategy, the processing speed of the first point cloud registration strategy is faster than that of the second point cloud registration strategy.

[0058] First, the overlap between the first point cloud data and the second point cloud data is calculated, and the target registration strategy is determined by comparing the overlap with the third preset value. If the overlap is greater than or equal to the third preset value, it means that the two point cloud data have a large overlap. At this time, the first point cloud registration strategy is selected as the target registration strategy, because the large overlap area makes it possible to complete the registration work well even if the first point cloud registration strategy with a slightly lower accuracy is used, and it has more advantages in terms of calculation speed. If the overlap is less than the third preset value, it means that the two point cloud data overlap less. In this case, a more accurate point cloud registration strategy is required to ensure the overall registration effect, so the second point cloud registration strategy with a higher accuracy is selected as the target registration strategy.

[0059] From the above, it can be concluded that when the overlap is high, the first point cloud registration strategy is adopted, which can use more coincidence information for rapid registration, improve processing efficiency, and save time and cost. Secondly, when the overlap is low, the second point cloud registration strategy is enabled to deal with complex situations with high precision and ensure registration accuracy. Therefore, this embodiment has strong adaptability, can be flexibly selected according to the actual situation of the data, optimize the registration process, and provide a reliable and efficient registration solution for three-dimensional point cloud processing in different fields.

[0060] In one embodiment of the present disclosure, the target registration strategy is a first point cloud registration strategy; The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data, including: Determine a first matrix based on the first point cloud data and the second point cloud data; The point cloud registration data is determined by performing calculations based on the eigenvectors of the first matrix.

[0061] In this embodiment, the point cloud registration data is fused data obtained after registration processing, which can accurately reflect the corresponding relationship between the first point cloud data and the second point cloud data in the same coordinate system and can be used for subsequent analysis, processing and application.

[0062] The first matrix is ​​a matrix obtained by specific calculation or transformation of the first point cloud data and the second point cloud data. The matrix contains certain association information between the above two point cloud data and is the basis for subsequent calculation of point cloud registration data.

[0063] The eigenvector of the first matrix refers to a non-zero vector whose direction remains unchanged or only undergoes a scaling transformation under the linear transformation corresponding to the matrix. In point cloud registration, the eigenvector of the first matrix can be used to extract the key features and direction information of the point cloud data to assist in determining the accurate registration relationship between point clouds.

[0064] This embodiment can use point cloud registration based on principal component analysis as the first point cloud registration strategy. Principal component analysis (PCA) is a data analysis technology, which is used for point cloud registration. Its core idea is to find the main components (i.e., the directions with the largest variance) in the point cloud registration data by eigendecomposing the covariance matrix of the point cloud registration data. In point cloud registration, the point cloud shape and direction information reflected by these main components can be used to determine the registration relationship.

[0065] In the registration process based on principal component analysis, the first point cloud data and the second point cloud data must first be combined to construct a covariance matrix (first matrix). For example, the point coordinate information in the two point cloud data is integrated and processed to calculate the covariance matrix, which reflects the distribution changes of the point cloud as a whole in space.

[0066] Perform eigendecomposition on the constructed covariance matrix to obtain its eigenvectors. Then, based on the main direction information of the point cloud represented by these eigenvectors, calculate the rotation and translation parameters between the two point clouds. For example, if an eigenvector represents a main axial direction of the point cloud, by comparing the difference in the axial direction between the two point cloud data, determine the angle of rotation and the distance of translation required, and then transform the first point cloud data to obtain the point cloud registration data after registration with the second point cloud data.

[0067] From the above, it can be concluded that this embodiment can effectively mine the intrinsic shape and direction information of the point cloud data, accurately determine the registration relationship, and improve the registration accuracy by constructing the first matrix and using its eigenvector to calculate the registration. Secondly, when processing point cloud data with a certain degree of overlap, the first point cloud registration strategy can give full play to its advantages, quickly complete the registration, and improve efficiency.

[0068] In one embodiment of the present disclosure, the target registration strategy is a second point cloud registration strategy; The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data, including: Determine an overlapping area between the first point cloud data and the second point cloud data; Determine matching feature points based on overlapping areas; Determine point cloud registration data based on matching feature points.

[0069] In this embodiment, the overlapping area refers to the portion where the first point cloud data and the second point cloud data cover and overlap each other in three-dimensional space. This area is critical for determining the correspondence between the first point cloud data and the second point cloud data, and is the basic range for subsequent searching for matching feature points.

[0070] In the overlapping area, points with corresponding relationships in the above two point cloud data are found through specific algorithms or feature extraction methods. These points usually have obvious geometric features or are unique in local areas, such as corner points, edge points, etc. Their matching relationship helps to determine the relative position and posture transformation of the two point cloud data.

[0071] This embodiment can use point cloud registration based on model-based geometric feature invariants as the second point cloud registration strategy. Geometric feature invariants refer to certain geometric features (such as the distance ratio and angle relationship between points) that remain unchanged when the object undergoes transformations such as translation, rotation, and scaling. By extracting and utilizing these invariants, matching relationships can be found between different point cloud data, thereby achieving registration.

[0072] This embodiment compares the coordinate ranges of each point in the two point cloud data to find out the parts that overlap in space and determine their boundaries and ranges. This is the basic step of alignment, and the subsequent feature point matching will be mainly performed in the overlapping area.

[0073] In the overlapping area, feature points are extracted according to the geometric feature invariants, and their corresponding relationship in the two point cloud data is determined by calculating the geometric relationship invariants between these feature points.

[0074] Afterwards, the coordinate information of the matching feature points in the two point cloud data is used to calculate the rotation and translation parameters required to transform the first point cloud data to align with the second point cloud data. For example, the transformation relationship between the matching feature points is fitted by a method such as the least squares method, and then the first point cloud data is rotated and translated according to the calculated parameters to obtain point cloud registration data.

[0075] From the above, it can be concluded that this embodiment can accurately locate the corresponding relationship between the first point cloud data and the second point cloud data by finding matching feature points, thereby greatly improving the registration accuracy. Secondly, this strategy focuses on overlapping areas, reduces unnecessary calculations, and improves registration efficiency. Furthermore, it has strong adaptability to point cloud data from different sources but with similar geometric structures. Whether it is design models and physical scan data, or data collected from different perspectives, it can be effectively registered, which enhances the versatility of point cloud registration in multiple fields and provides a reliable, efficient and accurate registration method for three-dimensional point cloud processing.

[0076] In one embodiment of the present disclosure, the 3D image reconstruction method further includes: If the registration result of the first point cloud data and the second point cloud data needs to meet the second condition, the third point cloud registration strategy is used as the target registration strategy; Among them, the accuracy of the third point cloud registration strategy is greater than the accuracy of the second point cloud registration strategy.

[0077] In this embodiment, the second condition is a pre-set standard setting for determining whether the registration is up to standard. The second accuracy may be greater than the first accuracy. In essence, it requires the registration to achieve a relatively higher level of accuracy, so as to measure whether the current registration result meets expectations and whether it can meet the requirements of subsequent three-dimensional image reconstruction for data accuracy.

[0078] The target registration strategy is the third point cloud registration strategy; The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data, including: Determine a first distance based on the first point cloud data and the second point cloud data; Iterate based on the first distance to determine point cloud registration data; If the iteration result is equal to the fourth preset value, the iteration stops; If the iteration result is less than the fourth preset value, the iteration continues.

[0079] The third point cloud registration strategy can be the iterative closest point method, which aims to find the optimal spatial transformation relationship between two different point cloud data through continuous iterative optimization, so that they can be aligned as accurately as possible in the same coordinate system. The core of this method is to perform repeated calculations and adjustments based on information such as the distance between point clouds.

[0080] The first distance is the distance measurement value between points determined by a specific calculation method in the iterative closest point method based on the coordinate information of each point in the first point cloud data and the second point cloud data. This distance can reflect the degree of difference between the two point cloud data in the current state and is an important basis for subsequent iterative adjustments. Common ones include Euclidean distance (i.e. the straight-line distance between two points in space).

[0081] Iteration refers to repeatedly executing the same or similar calculation steps according to certain rules and algorithms. Each iteration will make adjustments based on the results of the previous one. The purpose is to gradually optimize the registration effect between point cloud data and make them closer and closer to the state of accurate alignment. In the iterative closest point method, each iteration will update the transformation relationship between point clouds (such as rotation and translation parameters) based on the current distance and other information.

[0082] The fourth preset value is a pre-set threshold value used to determine whether the iteration process has achieved the expected goal and whether it can be stopped. It is usually related to the registration accuracy. For example, it can be a value representing the average distance between point clouds. When the distance-related index calculated during the iteration reaches this preset value, it is considered that the registration accuracy meets the requirements and the iteration can be terminated.

[0083] Taking the first distance calculated as a reference, the calculation is started in a loop according to the iteration rules set by the iterative closest point method. In each iteration, the closest point pairs in the two point cloud data are found based on the current distance information, and then the rotation and translation parameters that can better align the two point cloud data are calculated and updated based on the relationship between these point pairs. This process is repeated continuously to gradually optimize the registration effect until the accurately aligned point cloud registration data is finally determined.

[0084] After each iteration, a result related to the current registration status (such as the average distance between point clouds and other indicators) will be obtained, and this result will be compared with the pre-set first preset value. If the two are equal, it means that the current registration accuracy has reached the expected requirements, and the entire iteration process can be terminated, which means that point cloud registration data that meets the accuracy expectations has been obtained.

[0085] When the result obtained after each iteration (the relevant indicator reflecting the current degree of alignment) is less than the first preset value, it indicates that the current alignment effect has not reached the preset accuracy standard, and the two point cloud data need to be further adjusted and aligned. Therefore, the next round of iterative calculation should be continued according to the iteration rules, and the judgment and iteration process should be repeated continuously until the stopping condition is met.

[0086] From the above, it can be concluded that this embodiment flexibly switches the registration strategy according to the registration result, and enables the third point cloud registration strategy when the accuracy requirement is increased, which can ensure that the high-precision registration requirements are met. Secondly, the third point cloud registration strategy is based on distance iterative optimization, which can gradually and accurately adjust the point cloud data, effectively improve the registration accuracy, and reduce error accumulation. Furthermore, by judging the iterative stop condition through preset values, the entire registration process is controllable and efficient, avoiding excessive calculations, taking into account processing efficiency while ensuring accuracy, and enhancing the adaptability and reliability of three-dimensional image reconstruction in scenarios with different accuracy requirements.

[0087] Considering that there are both overlapping areas and non-overlapping areas between the first point cloud data and the second point cloud data, the fourth point cloud registration strategy can be used as a target registration strategy, wherein the fourth point cloud registration strategy is also a comprehensive registration strategy.

[0088] The target registration strategy is the fourth point cloud registration strategy; The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data, including: Determine an overlapping area of ​​the first point cloud data and the second point cloud data as a first area; Determine a non-overlapping area of ​​the first point cloud data and the second point cloud data as a second area; Using the first point cloud registration strategy as a first target registration strategy for the first region; Using the third point cloud registration strategy as the second target registration strategy for the second region; Processing the first point cloud data and the second point cloud data of the first area based on the first point cloud registration strategy to obtain first point cloud registration data; Processing the first point cloud data and the second point cloud data of the second area based on the second point cloud registration strategy to obtain second point cloud registration data; The first point cloud registration data and the second point cloud registration data are combined to obtain target point cloud data.

[0089] In this embodiment, the first point cloud registration strategy (such as point cloud registration based on principal component analysis) can use principal component analysis technology to perform feature decomposition on the covariance matrix of the data to find the main component directions in the first point cloud data and the second point cloud data in the first area, and then determine the transformation relationship between the point clouds based on these direction information. When applied in the first area, the main characteristic directions of the point clouds in the overlapping area can be quickly extracted, and the two point cloud data can be preliminarily aligned based on these directions. The advantage is that the calculation efficiency is relatively high, and a relatively reasonable registration result can be quickly obtained when the data quality requirements are not extremely stringent.

[0090] The third point cloud registration strategy (such as the iterative closest point method) is a high-precision registration method based on the idea of ​​continuous iterative optimization. When applied in the second area, by repeatedly calculating the closest point pairs between point clouds and updating the transformation matrix based on the relationship between these point pairs, the difference between the two point cloud data is gradually reduced to achieve extremely high registration accuracy.

[0091] Take point cloud registration in cultural relics restoration as an example: By using professional point cloud processing software to analyze the coordinate information of the two point cloud data, it was determined that a certain range around the neck and shoulders of the statue was an overlapping area (the first area), while the details of the statue's head (only in the second point cloud data) and most of the body (only in the first point cloud data) were non-overlapping areas (the second area).

[0092] For the first area (overlapping area), a point cloud registration strategy based on principal component analysis is adopted. The covariance matrix of the two point cloud data in this area is calculated, and the eigenvector is extracted to determine the main direction, such as the axial direction determined by the body posture of the statue, and the rotation and translation parameters are calculated to make the two point cloud data preliminarily aligned in this area to obtain the first point cloud registration data. For the second area (non-overlapping area), the iterative closest point method is used. First, the initial transformation parameters of the point cloud data of the statue's face are estimated, and then the nearest point pair is continuously searched. After multiple iterations, the facial detail point cloud data obtained by the handheld scanner and the body part point cloud data scanned at a long distance are highly accurately registered in the non-overlapping area to obtain the second point cloud registration data.

[0093] The first point cloud registration data and the second point cloud registration data are combined to form complete target point cloud data.

[0094] From the above, it can be concluded that the use of adaptation strategies for overlapping and non-overlapping areas of point cloud data can improve the overall efficiency and ensure the accuracy of key parts. Secondly, it can make full use of the advantages of different strategies and adapt to complex data conditions, which can effectively improve the quality and accuracy of the 3D image reconstruction model.

[0095] Corresponding to the three-dimensional image reconstruction method of the above embodiment, Figure 2 This is a block diagram of a 3D image reconstruction device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The three-dimensional image reconstruction device 20 includes: a point cloud data acquisition module 21, a demand determination module 22, a strategy determination module 23, a registration module 24 and a rendering module 25.

[0096] The point cloud data acquisition module 21 is used to acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources; A requirement determination module 22, configured to determine a target registration requirement based on the first point cloud data and the second point cloud data; A strategy determination module 23, configured to determine a target registration strategy from a plurality of point cloud registration strategies based on a target registration requirement, wherein the point cloud registration accuracies of the plurality of point cloud registration strategies are different; A registration module 24, configured to register the first point cloud data and the second point cloud data based on a target registration strategy to obtain point cloud registration data; The rendering module 25 is used to render the point cloud registration data to obtain a three-dimensional image model.

[0097] In one embodiment of the present disclosure, the point cloud data acquisition module 21 is specifically used to: In response to receiving a first captured image sent by a first device; First point cloud data is determined based on the first acquired image.

[0098] In one embodiment of the present disclosure, the point cloud data acquisition module 21 is further configured to: determining a plurality of characteristic values ​​according to the first acquired image; If a first eigenvalue among the multiple eigenvalues ​​is less than a first preset value, performing first image processing on the first collected image to obtain a first image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a first preset value, and the first eigenvalue is less than a second preset value, taking the first collected image as the second image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a second preset value, performing second image processing on the first collected image to obtain a third image; Determine first point cloud data based on the first image, the second image, or the third image; The first image processing and the second image processing are performed in different ways.

[0099] In one embodiment of the present disclosure, the strategy determination module 23 is specifically configured to: determining a degree of overlap between the first point cloud data and the second point cloud data; If the overlap is greater than or equal to the third preset value, the first point cloud registration strategy is used as the target registration strategy; If the overlap is less than the third preset value, the second point cloud registration strategy is used as the target registration strategy; Among them, the accuracy of the first point cloud registration strategy is less than the accuracy of the second point cloud registration strategy.

[0100] In one embodiment of the present disclosure, the target registration strategy is a first point cloud registration strategy; The registration module 24 is specifically used for: Determine a first matrix based on the first point cloud data and the second point cloud data; The point cloud registration data is determined by performing calculations based on the eigenvectors of the first matrix.

[0101] In one embodiment of the present disclosure, the target registration strategy is a second point cloud registration strategy; The registration module 24 is further configured to: The first point cloud data and the second point cloud data are registered based on the target registration strategy to obtain point cloud registration data, including: Determine an overlapping area between the first point cloud data and the second point cloud data; Determine matching feature points based on overlapping areas; Determine point cloud registration data based on matching feature points.

[0102] In one embodiment of the present disclosure, the policy determination module 23 is further configured to: If the registration result of the first point cloud data and the second point cloud data needs to meet the second condition, the third point cloud registration strategy is used as the target registration strategy; Among them, the accuracy of the third point cloud registration strategy is greater than the accuracy of the second point cloud registration strategy.

[0103] See also Figure 3 , Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 2 The functions of modules 21 to 25 are shown.

[0104] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0106] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0107] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the three-dimensional image reconstruction method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0108] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0109] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0114] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A three-dimensional image reconstruction method, characterized in that: include: Acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources; Determining target registration requirements based on the first point cloud data and the second point cloud data; Determining a target registration strategy from a plurality of point cloud registration strategies based on the target registration requirement, wherein the plurality of point cloud registration strategies have different point cloud registration accuracies; Registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data; The point cloud registration data is rendered to obtain a three-dimensional image model.

2. The three-dimensional image reconstruction method according to claim 1, characterized in that: The obtaining of the first point cloud data comprises: In response to receiving a first captured image sent by a first device; The first point cloud data is determined based on the first acquired image.

3. The three-dimensional image reconstruction method according to claim 2, characterized in that: The determining the first point cloud data based on the first acquired image includes: determining a plurality of feature values ​​according to the first acquired image; If a first eigenvalue among the multiple eigenvalues ​​is less than a first preset value, performing first image processing on the first collected image to obtain a first image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a first preset value, and the first eigenvalue is less than a second preset value, taking the first collected image as the second image; If a first eigenvalue among the multiple eigenvalues ​​is greater than or equal to a second preset value, performing second image processing on the first collected image to obtain a third image; Determine the first point cloud data based on the first image, the second image or the third image; The first image processing and the second image processing are performed in different ways.

4. The three-dimensional image reconstruction method according to claim 1, characterized in that: Determining a target registration strategy from a plurality of point cloud registration strategies based on the target registration requirement includes: determining a degree of overlap between the first point cloud data and the second point cloud data; If the overlap is greater than or equal to a third preset value, the first point cloud registration strategy is used as the target registration strategy; If the overlap is less than a third preset value, the second point cloud registration strategy is used as the target registration strategy; The accuracy of the first point cloud registration strategy is less than the accuracy of the second point cloud registration strategy.

5. The three-dimensional image reconstruction method according to claim 4, characterized in that: The target registration strategy is the first point cloud registration strategy; The registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data includes: Determine a first matrix according to the first point cloud data and the second point cloud data; The point cloud registration data is determined by performing calculation based on the eigenvector of the first matrix.

6. The three-dimensional image reconstruction method according to claim 4, characterized in that: The target registration strategy is the second point cloud registration strategy; The registering the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data includes: determining an overlapping area between the first point cloud data and the second point cloud data; Determining matching feature points based on the overlapping area; The point cloud registration data is determined based on the matching feature points.

7. The three-dimensional image reconstruction method according to claim 4, characterized in that: Also includes: If the registration result of the first point cloud data and the second point cloud data needs to meet the second condition, the third point cloud registration strategy is used as the target registration strategy; Among them, the accuracy of the third point cloud registration strategy is greater than the accuracy of the second point cloud registration strategy.

8. A three-dimensional image reconstruction device, characterized in that: include: A point cloud data acquisition module, used to acquire first point cloud data and second point cloud data; wherein the first point cloud data and the second point cloud data have different sources; A requirement determination module, configured to determine a target registration requirement based on the first point cloud data and the second point cloud data; A strategy determination module, configured to determine a target registration strategy from a plurality of point cloud registration strategies based on the target registration requirement, wherein the plurality of point cloud registration strategies have different point cloud registration accuracies; A registration module, configured to register the first point cloud data and the second point cloud data based on the target registration strategy to obtain point cloud registration data; The rendering module is used to render the point cloud registration data to obtain a three-dimensional image model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images