Three-dimensional model registration method and system, electronic equipment and storage medium

Abstract

The invention discloses a three-dimensional model registration method and system, electronic equipment and a storage medium, and the method comprises the steps: collecting images of a construction region in different time periods, and generating a three-dimensional model of the construction region in a corresponding time period based on the image of each time period, the three-dimensional model comprising a reference model and a to-be-registered model; in a first image used for generating a reference model and a second image used for generating a to-be-registered model, 2D feature matching points between the first image and the second image are extracted; according to the 2D feature matching points between the first image and the second image, determining 3D correlation points between the reference model and the to-be-registered model; and according to the 3D correlation points, determining a registration matrix between the reference model and the to-be-registered model, and based on the registration matrix, performing registration on the reference model and the to-be-registered model. The registration precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer-aided design, and in particular to a three-dimensional model registration method, system, electronic equipment and storage medium. Background Art

[0002] With the development of technology, images of the construction area at different times can be collected based on drone oblique photography technology. Based on the collected images, real-life models of the construction area at different times can be constructed. Based on the real-life models of the construction area at different times, the construction progress can be monitored and managed. For example, by collecting images of the construction area at the beginning of construction, a real-life model of the construction area can be constructed; by collecting images of the construction area at the middle of construction, a real-life model of the construction area can be constructed. After the real-life models at the beginning of construction and the middle of construction are overlapped and compared, the amount of construction completed from the beginning of construction to the middle of construction can be intuitively viewed. However, for this kind of real-life model constructed based on images, there may be deviations between the models. For example, in the real-life model constructed for the first time, the target object A may be located at position A, and in the real-life model constructed for the second time, the target object A may be located at position B with a certain distance from position A, which is not conducive to performing related operations such as overlap comparison on the real-life model. Therefore, it is necessary to align different real-life models.

[0003] At present, some technologies are based on 3D registration algorithms (such as iterative closest point ICP, generalized iterative closest point GICP, and normal distribution point cloud transformation NDT) to align different real scene models. The registration accuracy of this method is not high. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a three-dimensional model registration method, a three-dimensional model registration system, an electronic device and a computer-readable storage medium, which can improve the registration accuracy.

[0005] In one aspect, the present invention provides a three-dimensional model registration method, the method comprising:

[0006] Collect images of the construction area at different time periods, and generate a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered;

[0007] In a first image used to generate the reference model and a second image used to generate the model to be registered, extracting 2D feature matching points between the first image and the second image;

[0008] Determining 3D association points between the reference model and the model to be registered according to 2D feature matching points between the first image and the second image;

[0009] A registration matrix between the reference model and the model to be registered is determined according to the 3D associated points, and the reference model and the model to be registered are registered based on the registration matrix.

[0010] In the technical solutions of some embodiments of the present application, a three-dimensional model of the construction area at the corresponding time period is generated based on the images of the construction area at each time period, and the 3D correlation points between the three-dimensional models are obtained based on the 2D feature matching points between the images. The 3D correlation points obtained in this way have high accuracy. In this way, the registration matrix determined based on the 3D correlation points has high accuracy, thereby improving the registration accuracy.

[0011] In some embodiments, determining the 3D connection point between the reference model and the model to be registered includes:

[0012] Projecting the 2D feature matching points in the first image to the first coordinate system where the reference model is located, and taking the intersection of the projected points and the reference model to obtain a first 3D feature point set;

[0013] Projecting the 2D feature matching points in the second image to the second coordinate system where the reference model is located, and taking the intersection of the projected points and the model to be registered to obtain a second 3D feature point set;

[0014] The 3D associated points are determined in the first 3D feature point set and the second 3D feature point set.

[0015] By projecting the 2D feature matching points into the first coordinate system and the second coordinate system, the 3D associated points of the reference model and the model to be calibrated are obtained. The 3D associated points obtained in this way have high accuracy.

[0016] In some embodiments, when the first coordinate system and the second coordinate system are the same, determining the 3D correlation point between the reference model and the model to be registered includes:

[0017] For a first 3D feature point in the first 3D feature point set and a second 3D feature point in the second 3D feature point set, if it is determined that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range based on coordinate information of the first 3D feature point and the second 3D feature point, the first 3D feature point and the second 3D feature point are determined as the 3D associated points.

[0018] In this way, abnormal 3D correlation points can be eliminated to obtain 3D correlation points with stronger robustness and higher accuracy.

[0019] In some embodiments, when the first coordinate system and the second coordinate system are different, determining the 3D correlation point between the reference model and the model to be registered includes:

[0020] For a first 3D feature point in the first 3D feature point set and a second 3D feature point in the second 3D feature point set, convert the first 3D feature point and the second 3D feature point into the same target coordinate system;

[0021] If it is determined that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range according to the coordinate information of the first 3D feature point and the second 3D feature point in the target coordinate system, the first 3D feature point and the second 3D feature point are determined as the 3D associated points.

[0022] Through the coordinate system conversion, the first 3D feature point and the second 3D feature point can be located in the same coordinate system, thereby ensuring the correctness of the calculated distance.

[0023] In some embodiments, projecting the 2D feature matching points in the first image to the first coordinate system, or projecting the 2D feature matching points in the second image to the second coordinate system, includes:

[0024] According to the internal and external parameters of the camera for image acquisition, a ray is drawn from the camera center point of the camera through the 2D feature matching point, and the 2D feature matching point is projected into the Earth-centered Earth-fixed coordinate system;

[0025] The projection point of the 2D feature matching point in the Earth-centered Earth-fixed coordinate system is converted to the corresponding first coordinate system or second coordinate system.

[0026] The scheme has good feasibility, and the projection points of the obtained 2D feature matching points in the first coordinate system are relatively accurate.

[0027] In some embodiments, extracting 2D feature matching points between the first image and the second image includes:

[0028] Extracting a first 2D feature point from the first image, and generating a first descriptor of the first 2D feature point;

[0029] Extracting second 2D feature points from the second image, and generating second descriptors of the second 2D feature points;

[0030] If the similarity between the first descriptor and the second descriptor exceeds a similarity threshold, the first 2D feature point and the second 2D feature point are used as the 2D feature matching points.

[0031] By extracting feature points from the first image and the second image, and then determining 2D feature matching points based on the similarity of descriptors of the feature points, the obtained 2D feature matching points have high accuracy.

[0032] In some embodiments, after extracting the 2D feature matching points, the method further includes:

[0033] According to the distortion parameters of the camera for image acquisition, the coordinate information of the 2D feature matching points is subjected to distortion correction.

[0034] In this way, the accuracy of 2D feature matching points can be improved.

[0035] Another aspect of the present invention further provides a three-dimensional model registration system, the system comprising:

[0036] A model building module, used to collect images of the construction area at different time periods, and generate a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered;

[0037] A 2D feature matching module, used to determine 2D feature matching points between a first image used to generate the reference model and a second image used to generate the model to be registered;

[0038] A 3D association module, used to determine 3D association points between the reference model and the model to be registered based on 2D feature matching points between the first image and the second image;

[0039] A registration module is used to determine a registration matrix between the reference model and the model to be registered according to the 3D associated points, and adjust the model to be registered based on the registration matrix.

[0040] Another aspect of the present invention provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.

[0041] Another aspect of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0043] Figure 1A schematic diagram of a process of a three-dimensional model registration method provided by an embodiment of the present application is shown;

[0044] Figure 2 A schematic diagram of a process for determining 3D associated points provided by an embodiment of the present application is shown;

[0045] Figure 3 A schematic diagram of modules of a three-dimensional model registration system provided by an embodiment of the present application is shown;

[0046] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] The present application provides a 3D model registration method, which can improve the registration accuracy. The 3D model configuration method can be applied to electronic devices. The electronic devices include but are not limited to laptop computers, desktop computers, smart phones, tablet computers, etc. Figure 1 , which is a flow chart of a three-dimensional model registration method provided in one embodiment of the present application. Figure 1 In the present invention, the 3D model registration method includes the following steps:

[0049] Step S11, collecting images of the construction area at different time periods, and generating a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered.

[0050] In some embodiments, based on the oblique photography technology of unmanned aerial vehicles, unmanned aerial vehicles with high-precision positioning equipment can be used to collect images of the construction area at different time periods. Among them, different time periods can refer to different construction time periods, such as the early construction period, the middle construction period, the late construction period, etc. In each time period, multiple images of the construction area can be collected. These multiple images can be images of different positions in the construction area, or they can be images of the same position collected from different angles. The images collected at different time periods need to have overlapping areas. For example, the images collected in each time period include images of sub-area A of the construction area. At the same time, when collecting images, periods with bad weather such as thunderstorms can be avoided to prevent the quality of the collected images from being too poor and affecting the accuracy of the three-dimensional model registration.

[0051] In some embodiments, the images collected at each time period can be input into the 3D reconstruction software respectively, and the 3D reconstruction software performs aerial triangulation and dense reconstruction to generate a 3D model of the construction area at the corresponding time period. For example, the images collected at the early stage of construction are input into the 3D reconstruction software to obtain the 3D model of the construction area at the early stage of construction; the images collected at the middle stage of construction are input into the 3D reconstruction software to obtain the 3D model of the construction area at the middle stage of construction.

[0052] The 3D reconstruction software can generate corresponding aerial triangulation solution files and OBJ files (files in obj format) every time a 3D model is created.

[0053] Among them, the aerial triangulation file is generated during the aerial triangulation solution process, and mainly includes the internal and external parameters of the camera (i.e., drone) used for image acquisition. The camera's internal parameters may include the camera's focal length, principal point, distortion parameters, etc. The camera's external parameters may include the rotation matrix from the Earth-Centered, Earth-Fixed (ECEF) coordinate system to the camera coordinate system, the 3D position of the camera's center point in the Earth-Centered, Earth-Fixed coordinate system, etc. By parsing the aerial triangulation file, the camera's internal and external parameters can be obtained.

[0054] OBJ files are generated during dense reconstruction, and mainly include vertex data of the created 3D model, such as vertex coordinates, texture coordinates, vertex normal vectors, etc. OBJ files can be understood as description files of 3D models. By parsing OBJ files, data information of 3D models can be obtained.

[0055] In some embodiments, in the three-dimensional models of each time period created, the reference model and the model to be calibrated can be specified by manual selection. The reference model can refer to a three-dimensional model with higher accuracy, and the model to be calibrated can refer to a three-dimensional model with lower accuracy. By rotating or translating the model to be calibrated, the model to be calibrated can be aligned with the reference model, that is, the reference model and the model to be calibrated are registered. The following describes the registration process of the reference model and the model to be calibrated through steps S12 to S14.

[0056] Step S12: extracting 2D feature matching points between the first image used to generate the reference model and the second image used to generate the model to be registered.

[0057] Among them, the 2D feature matching points can be similar or identical feature points in the first image and the second image. These similar or identical feature points can correspond to the same point in the construction area. Specifically, the 2D feature matching points can be feature points in the overlapping area of ​​the first image and the second image. For example, suppose there is a building A in the construction area. When the first image and the second image are captured, the image of building A (i.e., the overlapping area) is captured, then the points on building A in the first image and the points on building A in the second image can constitute 2D feature matching points.

[0058] In this embodiment, 2D feature matching points between the first image and the second image can be extracted based on Sift feature extraction and feature matching technology. Specifically, the first 2D feature point can be extracted from the first image, and a first descriptor of the first 2D feature point can be generated, and the second 2D feature point can be extracted from the second image, and a second descriptor of the second 2D feature point can be generated. If the similarity between the first descriptor and the second descriptor exceeds a similarity threshold, the first 2D feature point and the second 2D feature point are used as 2D feature matching points.

[0059] The first 2D feature point may be a feature point extracted from a first image in a different scale space, and the second 2D feature point may be a feature point extracted from a second image in a different scale space. Images in different scale spaces (i.e., the first image or the second image) may be understood as images of different sizes.

[0060] The first descriptor can characterize the size, direction, scale and other information of the first 2D feature point, and the second descriptor can characterize the size, direction, scale and other information of the second 2D feature point. The similarity between the first descriptor and the second descriptor can be determined based on the distance between the first descriptor and the second descriptor. The smaller the distance, the higher the similarity between the first descriptor and the second descriptor. If the similarity between the first descriptor and the second descriptor exceeds the similarity threshold, the first 2D feature point and the second 2D feature point can be used as 2D feature matching points.

[0061] The 2D feature matching points can be saved in a feature matching file. The feature matching file can include coordinate information of the 2D feature matching points. In this way, in subsequent operations, the 2D feature matching points between the first image and the second image can be obtained based on the feature matching file. By extracting feature points from the first image and the second image in different scale spaces, and then determining the 2D feature matching points based on the similarity of the descriptors of the feature points, the obtained 2D feature matching points have high accuracy.

[0062] In some other embodiments, 2D feature matching points may also be extracted based on feature descriptors such as SURF, ORB, HAHOG, etc. These technologies are conventional technologies that those skilled in the art should know and will not be described in detail here.

[0063] In some embodiments, taking into account the reasons such as the camera lens shape, the lens installation position, and the lens design defects, the coordinate information of the 2D feature matching points may cause errors, that is, the coordinate information of the 2D feature matching points in the image deviates from the theoretical coordinate information of the 2D feature matching points (i.e., distorted). In order to solve this problem, after the 2D feature matching points are extracted, the coordinate information of the 2D feature matching points may be subjected to distortion correction according to the distortion parameters of the camera used for image acquisition, so that the coordinate information of the 2D feature matching points in the image is the same as the theoretical coordinate information of the 2D feature matching points. In this way, the accuracy of the 2D feature matching points may be improved. When performing distortion correction on the coordinate information of the 2D feature matching points, some conventional technical means in the field may be used, which will not be elaborated in this application.

[0064] After the above-mentioned operations in step S12, 2D feature matching points with high accuracy can be extracted. It can be understood that multiple groups of 2D feature matching points can be extracted from the first image and the second image.

[0065] Step S13, determining 3D associated points between the reference model and the model to be registered based on the 2D feature matching points between the first image and the second image.

[0066] Specifically, similar to 2D feature matching points, 3D association points can be similar or identical feature points in the reference model and the model to be registered. These similar or identical feature points can correspond to the same point in the construction area. In other words, 2D feature matching points are similar or identical feature points extracted from different two-dimensional images (i.e., the first image and the second image), and 3D association points are similar or identical feature points extracted from different three-dimensional models (i.e., the reference model and the model to be calibrated). 3D association points can be points converted from 2D feature matching points. For example, assume that the first 2D feature point A in the first image and the second 2D feature point B in the second image are 2D feature matching points. After the reference model is generated from the first image, the first 2D feature point A is converted to a 3D feature point A in the reference model; after the model to be calibrated is generated from the second image, the second 2D feature point B is converted to a 3D feature point B in the model to be calibrated. Then, 3D feature point A and 3D feature point B constitute 3D association points.

[0067] Based on the above description, please refer to Figure 2 , which is a schematic diagram of a process for determining 3D associated points provided by an embodiment of the present application. Figure 2 The process may include the following steps:

[0068] Step S21, projecting the 2D feature matching points in the first image to the first coordinate system where the reference model is located, and taking the intersection of the projected points and the reference model to obtain a first 3D feature point set.

[0069] Among them, projecting the 2D feature matching points in the first image to the first coordinate system means converting the 2D feature matching points in the first image to the first coordinate system through coordinate transformation based on the internal and external parameters of the camera. In this embodiment, considering that the external parameters of the camera include the rotation matrix from the geocentric earth-fixed coordinate system to the camera coordinate system and the 3D position of the center point of the camera in the geocentric earth-fixed coordinate system, the above-mentioned projecting the 2D feature matching points in the first image to the first coordinate system where the reference model is located may include:

[0070] According to the internal and external parameters of the camera for image acquisition, a ray is drawn from the camera center point through the 2D feature matching point, and the 2D feature matching point is projected into the Earth-centered Earth-fixed coordinate system;

[0071] The projection point of the 2D feature matching point in the Earth-centered Earth-fixed coordinate system is converted to the first coordinate system.

[0072] Among them, the above-mentioned ray can represent the direction vector from the center point of the camera passing through the 2D feature matching point. The direction of the ray is determined based on the internal and external parameters of the camera. The ray can point to a point in the geocentric earth-fixed coordinate system, which can be the projection point of the 2D feature matching point in the geocentric earth-fixed coordinate system. Furthermore, the coordinate system definition of the first coordinate system can be obtained from the OBJ file corresponding to the reference model. At the same time, the coordinate system definition of the geocentric earth-fixed coordinate system is also well known. Based on the coordinate system definition of the first coordinate system and the coordinate system definition of the geocentric earth-fixed coordinate system, the transformation matrix between the two coordinate systems can be known. Based on the transformation matrix between the two coordinate systems, the projection point of the 2D feature matching point in the geocentric earth-fixed coordinate system can be converted to the first coordinate system to obtain the projection point of the 2D feature matching point in the first coordinate system.

[0073] This application is based on the internal and external parameters of the camera. By drawing a ray from the center point of the camera through the 2D feature matching point, the projection point of the 2D feature matching point in the Earth-centered Earth-fixed coordinate system is obtained, and then the projection point of the 2D feature matching point in the first coordinate system is obtained by conversion. The solution is specific and has good feasibility, and the projection point of the 2D feature matching point in the first coordinate system is relatively accurate.

[0074] In some other embodiments, if the external parameters of the camera directly specify the rotation matrix from the first coordinate system to the camera coordinate system, it is also possible to directly project the 2D feature matching points to the first coordinate system by drawing a ray from the camera center through the 2D feature matching points according to the internal and external parameters of the camera. That is, there is no need to project to the Earth-centered Earth-fixed coordinate system. In this way, the projection process can be simplified.

[0075] It is understandable that the first image may include multiple 2D feature matching points. After projecting these 2D feature matching points to the first coordinate system, a projection point set can be obtained. By intersecting the projection point set with the reference model, the first 3D feature point set can be obtained. Specifically, since the OBJ file of the reference model is used to describe the reference model, the projection point set can be intersected with the OBJ file of the reference model to obtain the first 3D feature point set. The first 3D feature point set may include multiple feature points, which are the feature points corresponding to the reference model and the 2D feature matching points in the first image.

[0076] Step S22, projecting the 2D feature matching points in the second image to the second coordinate system where the reference model is located, and taking the intersection of the projected points and the model to be registered to obtain a second 3D feature point set.

[0077] Step S22 is basically similar to step S21 and will not be described in detail here.

[0078] Step S23: 3D associated points may be determined in the first 3D feature point set and the second 3D feature point set.

[0079] Specifically, since the feature points in the first 3D feature point set and the second 3D feature point set are obtained based on the 2D feature matching points, for any feature point A in the first 3D feature point set, a corresponding feature point B can be found in the second 3D feature point set, so that the feature point A and the feature point B constitute a 3D associated point. However, there may be abnormal points in these 3D associated points. For example, in the process of building a three-dimensional model, the 3D associated point corresponding to the 2D feature matching point is placed in the wrong position, which causes the 3D associated point to be abnormal. In view of this, this embodiment can screen and eliminate the abnormal 3D associated points based on the distance between the 3D associated points to obtain 3D associated points with stronger robustness and higher precision. Among them, the distance between the 3D associated points refers to the distance between the 3D associated points in the same coordinate system. If the feature point A under the reference model and the feature point B of the model to be constructed constitute a 3D associated point, but the distance between the two feature points is too large or too small, both feature points can be considered to be abnormal 3D associated points and need to be eliminated.

[0080] Furthermore, considering that the 3D reconstruction software will determine the coordinate system of the created 3D model according to the actual situation when constructing the 3D model, that is, the first coordinate system where the reference model is located and the second coordinate system where the model to be calibrated is located may be the same or different. Therefore, when judging the distance between 3D associated points, it is necessary to first determine whether the first coordinate system where the reference model is located and the second coordinate system where the model to be calibrated is located are the same. When the first coordinate system and the second coordinate system are the same, determining the 3D associated points between the reference model and the model to be registered may include:

[0081] For the first 3D feature point in the first 3D feature point set and the second 3D feature point in the second 3D feature point set, if it is determined based on the coordinate information of the first 3D feature point and the second 3D feature point that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range, the first 3D feature point and the second 3D feature point are determined as 3D associated points.

[0082] In this way, abnormal 3D correlation points can be eliminated to obtain 3D correlation points with stronger robustness and higher accuracy.

[0083] In the case where the first coordinate system and the second coordinate system are different, determining the 3D associated points between the reference model and the model to be registered may include:

[0084] For a first 3D feature point in the first 3D feature point set and a second 3D feature point in the second 3D feature point set, convert the first 3D feature point and the second 3D feature point into the same target coordinate system;

[0085] If it is determined according to coordinate information of the first 3D feature point and the second 3D feature point in the target coordinate system that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range, the first 3D feature point and the second 3D feature point are determined as 3D associated points.

[0086] In this embodiment, the target coordinate system is the second coordinate system, that is, the first 3D feature point and the second 3D feature point are both converted to the second coordinate system. In this way, through the coordinate system conversion, the first 3D feature point and the second 3D feature point can be located in the same coordinate system, ensuring the accuracy of the calculated distance.

[0087] In some embodiments, the first coordinate system where the reference model is located can be queried in the OBJ file corresponding to the reference model, and the second coordinate system where the model to be calibrated is located can be queried in the OBJ file where the model to be calibrated is located. Based on the query results, it can be determined whether the first coordinate system and the second coordinate system are the same.

[0088] In some embodiments, considering that the distance between the first 3D feature point and the second 3D feature point conforms to the normal distribution, the above-mentioned distance threshold range can be determined based on the normal distribution of the distance. For example, the distance distributed between (μ-3б, μ+3б) is determined as the distance threshold range.

[0089] In some other embodiments, 3D associated points with too large or too small distances may be removed based on radius filtering, Gaussian filtering, conditional filtering, etc. using area blocks.

[0090] The present application obtains 3D associated points of the reference model and the model to be calibrated by projecting 2D feature matching points into the first coordinate system and the second coordinate system. The 3D associated points obtained in this way have high accuracy.

[0091] Step S14, determining a registration matrix between the reference model and the model to be registered according to the 3D associated points, and registering the reference model and the model to be registered based on the registration matrix.

[0092] Specifically, the registration matrix can characterize the angle and displacement that the model to be registered needs to rotate when aligning the model to be registered with the reference model. The registration matrix can be determined based on techniques that those skilled in the art should know, such as the LM method. Based on the relevant description of step S13, it can be understood that when the reference model and the model to be registered are located in the same coordinate system, the registration matrix can represent the offset between the reference model and the model to be registered. When the reference model and the model to be registered are located in different coordinate systems, the registration matrix can represent the conversion matrix between the first coordinate system where the reference model is located and the second coordinate system where the model to be registered is located, as well as the offset between the reference model and the model to be registered.

[0093] After obtaining the registration matrix, the model to be registered can be transformed according to the registration matrix, and the rmse value of the distance between the converted model to be registered and the reference model can be calculated. If the rmse value is too large, it means that there is still a large deviation between the converted model to be registered and the reference model (that is, not completely aligned). You can continue to select more 3D associated points and recalculate the registration matrix; if the rmse value is within the specified range, it means that the deviation between the converted model to be registered and the reference model is small, and you can continue to perform subsequent related operations, such as aligning the reference model and the model to be registered, and checking the construction progress of the construction area at different times.

[0094] In summary, in the technical solutions of some embodiments of the present application, a three-dimensional model of the construction area at the corresponding time period is generated based on the images of the construction area at each time period, and the 3D correlation points between the three-dimensional models are obtained based on the 2D feature matching points between the images. The 3D correlation points obtained in this way have high accuracy. In this way, the registration matrix determined based on the 3D correlation points has high accuracy, thereby improving the registration accuracy.

[0095] In addition, compared with the manual specification of 3D association points in some technologies, the present application can automatically determine the 3D association points, reducing the workload of manually specifying 3D association points and being more efficient.

[0096] See also Figure 3 , is a schematic diagram of a module of a three-dimensional model registration system provided by an embodiment of the present application. The three-dimensional model registration system includes:

[0097] A model building module, used to collect images of the construction area at different time periods, and generate a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered;

[0098] A 2D feature matching module, used to determine 2D feature matching points between a first image used to generate the reference model and a second image used to generate the model to be registered;

[0099] A 3D association module, used to determine 3D association points between the reference model and the model to be registered based on 2D feature matching points between the first image and the second image;

[0100] A registration module is used to determine a registration matrix between the reference model and the model to be registered according to the 3D associated points, and adjust the model to be registered based on the registration matrix.

[0101] See also Figure 4 , is a schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0102] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0103] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0104] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] One embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above-mentioned method for associating cross-sectional views is implemented.

[0106] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A three-dimensional model registration method, characterized in that: The method comprises: Collect images of the construction area at different time periods, and generate a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered; In a first image used to generate the reference model and a second image used to generate the model to be registered, extracting 2D feature matching points between the first image and the second image; Determining 3D association points between the reference model and the model to be registered according to 2D feature matching points between the first image and the second image; A registration matrix between the reference model and the model to be registered is determined according to the 3D associated points, and the reference model and the model to be registered are registered based on the registration matrix.

2. The method according to claim 1, characterized in that The determining of the 3D associated points between the reference model and the model to be registered comprises: Projecting the 2D feature matching points in the first image to the first coordinate system where the reference model is located, and taking the intersection of the projected points and the reference model to obtain a first 3D feature point set; Projecting the 2D feature matching points in the second image to the second coordinate system where the reference model is located, and taking the intersection of the projected points and the model to be registered to obtain a second 3D feature point set; The 3D associated points are determined in the first 3D feature point set and the second 3D feature point set.

3. The method according to claim 2, characterized in that In the case where the first coordinate system and the second coordinate system are the same, determining the 3D associated points between the reference model and the model to be registered includes: For a first 3D feature point in the first 3D feature point set and a second 3D feature point in the second 3D feature point set, if it is determined that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range based on coordinate information of the first 3D feature point and the second 3D feature point, the first 3D feature point and the second 3D feature point are determined as 3D associated points.

4. The method according to claim 2, characterized in that In the case that the first coordinate system and the second coordinate system are different, determining the 3D associated points between the reference model and the model to be registered includes: For a first 3D feature point in the first 3D feature point set and a second 3D feature point in the second 3D feature point set, convert the first 3D feature point and the second 3D feature point into the same target coordinate system; If it is determined that the distance between the first 3D feature point and the second 3D feature point is within a distance threshold range according to the coordinate information of the first 3D feature point and the second 3D feature point in the target coordinate system, the first 3D feature point and the second 3D feature point are determined as 3D associated points.

5. The method according to claim 2, characterized in that Projecting the 2D feature matching points in the first image to the first coordinate system, or projecting the 2D feature matching points in the second image to the second coordinate system, includes: According to the internal and external parameters of the camera for image acquisition, a ray is drawn from the camera center point of the camera through the 2D feature matching point, and the 2D feature matching point is projected into the Earth-centered Earth-fixed coordinate system; The projection point of the 2D feature matching point in the Earth-centered Earth-fixed coordinate system is converted to the corresponding first coordinate system or second coordinate system.

6. The method according to claim 1, characterized in that The extracting 2D feature matching points between the first image and the second image includes: Extracting a first 2D feature point from the first image, and generating a first descriptor of the first 2D feature point; Extracting second 2D feature points from the second image, and generating second descriptors of the second 2D feature points; If the similarity between the first descriptor and the second descriptor exceeds a similarity threshold, the first 2D feature point and the second 2D feature point are used as the 2D feature matching points.

7. The method according to claim 6, characterized in that After extracting the 2D feature matching points, the method further includes: According to the distortion parameters of the camera for image acquisition, the coordinate information of the 2D feature matching points is subjected to distortion correction.

8. A three-dimensional model registration system, characterized in that: The system comprises: A model building module, used to collect images of the construction area at different time periods, and generate a three-dimensional model of the construction area at the corresponding time period based on the images of each time period, wherein the three-dimensional model includes a reference model and a model to be registered; A 2D feature matching module, used to determine 2D feature matching points between a first image used to generate the reference model and a second image used to generate the model to be registered; A 3D association module, used to determine 3D association points between the reference model and the model to be registered based on 2D feature matching points between the first image and the second image; A registration module is used to determine a registration matrix between the reference model and the model to be registered according to the 3D associated points, and adjust the model to be registered based on the registration matrix.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images