Point cloud construction method and device, electronic equipment and readable storage medium
By estimating camera attitude information and determining binocular image pairs, dense point clouds are built, and the low rendering accuracy caused by sparse point clouds in the prior art is solved, which significantly improves the three-dimensional rendering effect.
Patent Information
- Application Number
- CN202411115772.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-29
AI Technical Summary
In the existing three-dimensional rendering technology, sparse point clouds built on SfM technology lead to low accuracy and poor quality of rendered scenes.
By acquiring at least two image frames and camera parameters of the camera, estimating camera posture information, determining binocular image pairs that meet binocular constraints, and stereo matching is performed to build dense point clouds.
Improves the geometric description accuracy and quality of the rendering scene and improves the rendering effect of the three-dimensional scene.
Smart Images

Figure CN119048675A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a point cloud construction method, device, electronic device and readable storage medium. Background Art
[0002] 3D rendering refers to the process of adding skin, texture and other elements to the established 3D model through rendering, and performing material mapping and other processing. Generally, rendering can make up for the shortcomings of 3D modeling, surface the complex modeling, and make the 3D model present a 2D image with high realism in the 3D model grid.
[0003] In the related art, three-dimensional rendering is usually achieved by the following method: multiple image frames of a certain scene are captured, and then, based on the Structure-from-Motion (SfM) technology, point clouds are constructed for the multiple image frames to obtain point clouds corresponding to the multiple image frames, and then, the multiple image frames are three-dimensionally rendered based on the constructed point clouds.
[0004] However, in the above rendering process, since the point cloud constructed using the SfM technology is usually a sparse point cloud with a small number of point clouds, the geometric description accuracy of the rendered scene is low, so the rendered scene has problems such as low accuracy and poor quality. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a point cloud construction method, device, electronic device and readable storage medium, which can construct a dense point cloud with a larger number of point clouds, thereby improving the accuracy and quality of the geometric description of the rendered scene.
[0006] In a first aspect, an embodiment of the present application provides a point cloud construction method, the method comprising: acquiring at least two image frames and camera parameters of a camera that captures at least two image frames; estimating first camera posture information based on at least two image frames and the camera parameters, the first camera posture information comprising camera posture information corresponding to each image frame; based on the first camera posture information, determining at least one group of binocular image pairs that satisfy binocular constraint conditions from at least two image frames; performing stereo matching on at least one group of binocular image pairs to construct a dense point cloud corresponding to at least two image frames.
[0007] In a second aspect, an embodiment of the present application provides a point cloud construction device, which includes: an acquisition module, a processing module and a determination module. The acquisition module is used to acquire at least two image frames and camera parameters of a camera that captures at least two image frames. The processing module is used to estimate first camera posture information based on at least two image frames and camera parameters acquired by the acquisition module, and the first camera posture information includes camera posture information corresponding to each image frame. The determination module is used to determine at least one group of binocular image pairs that meet binocular constraint conditions from at least two image frames based on the first camera posture information obtained by the estimation module. The processing module is also used to perform stereo matching on at least one group of binocular image pairs determined based on the determination module to construct a dense point cloud corresponding to at least two image frames.
[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0012] In an embodiment of the present application, at least two image frames and camera parameters of a camera that captures at least two image frames are obtained; based on at least two image frames and camera parameters, first camera posture information is estimated, and the first camera posture information includes camera posture information corresponding to each image frame; based on the first camera posture information, at least one group of binocular image pairs that meet binocular constraints are determined from at least two image frames; stereo matching is performed on at least one group of binocular image pairs to construct a dense point cloud corresponding to at least two image frames. In this solution, since the first camera posture information is used to characterize the acquisition trajectory of the camera to acquire multiple image frames, based on the first camera posture information, an image pair that approximates an ideal binocular image pair can be selected from the multiple image frames, and then, the selected binocular image pair is used to reconstruct a three-dimensional dense point cloud, so that a dense point cloud with a larger number of point clouds can be constructed, so that when the electronic device performs three-dimensional scene rendering based on the dense point cloud, the geometric description accuracy and quality of the rendered scene can be improved, thereby improving the rendering effect of the three-dimensional scene. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is one of the flow charts of a point cloud construction method provided in an embodiment of the present application;
[0014] Figure 2 This is the second flow chart of a point cloud construction method provided in an embodiment of the present application;
[0015] Figure 3 This is a third flow chart of a point cloud construction method provided in an embodiment of the present application;
[0016] Figure 4 This is a fourth flow chart of a point cloud construction method provided in an embodiment of the present application;
[0017] Figure 5 This is a fifth flow chart of a point cloud construction method provided in an embodiment of the present application;
[0018] Figure 6 is a schematic diagram of a point cloud construction device provided in an embodiment of the present application;
[0019] Figure 7 is a schematic diagram of an electronic device provided in an embodiment of the present application;
[0020] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0023] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".
[0024] The following is an explanation of some concepts or terms involved in the point cloud construction method, device, electronic device and readable storage medium provided in the embodiments of the present application.
[0025] Structure-from-Motion (SfM): It aims to recover the structural information of the scene and the pose of the camera from multiple images of different perspectives. It is one of the key technologies of computer 3D vision.
[0026] Bundle Adjustment (BA): It is the optimization step in SfM, which optimizes the camera intrinsic parameters, camera posture, and three-dimensional position of feature points by minimizing the reprojection error.
[0027] Simultaneous Localization and Mapping (SLAM): Mainly used to solve the positioning and mapping problems of mobile sensors when moving in unknown environments.
[0028] Neural Radiance Field (NeRF): is a neural network that can reconstruct complex 3D scenes from partial 2D images.
[0029] 3D Gaussian Splatting (3DGS): is an efficient rendering technique that can generate high-quality surface details and shadow effects.
[0030] Scale Invariant Feature Transform (SIFT): is an algorithm used to detect and describe local features in digital images.
[0031] Oriented FAST and Rotated BRIEF (ORB): It is the abbreviation of Oriented Fast and Rotated Brief. It can be used to quickly create feature vectors for key points in an image. These feature vectors can be used to identify objects in the image.
[0032] Random Sample Consensus (RANSAC): is a robust estimation method that was first used for pose estimation problems in computer vision and is now widely used in parameter estimation problems of known models.
[0033] Special Euclidean (3) Group: is a Lie group used to represent posture transformations in three-dimensional space.
[0034] The point cloud construction method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0035] Since its introduction, Neural Radiance Field (NeRF) has been widely used in multiple applications such as new perspective rendering and scene 3D reconstruction. Specifically, when completing 3D model rendering based on NeRF, it is necessary to first use the Structure-from-Motion (SfM) technology to perform camera pose estimation and sparse point cloud reconstruction, and then initialize the 3D Gaussian through the reconstructed sparse point cloud to construct the 3D model corresponding to the sparse point cloud. Finally, the network model based on the 3D Gaussian Splatting (3DGS) technology in NeRF is used to render the 3D model. It should be noted that in the above rendering process, the accuracy and speed of camera pose estimation and sparse point cloud reconstruction using SfM technology have an important impact on the optimization effect of the subsequent rendering process using 3DGS technology.
[0036] Among them, the above NeRF is a neural network that can reconstruct complex three-dimensional scenes from partial two-dimensional images. The above SfM technology aims to recover the structural information of the scene and the pose of the camera from multiple images of different perspectives, and is one of the key technologies of computer three-dimensional vision. The above 3DGS technology is an efficient rendering technology that can generate high-quality surface details and shadow effects.
[0037] In the above rendering process, since the point cloud constructed using the SfM technology is usually a sparse point cloud with a small number of point clouds, the geometric description accuracy of the rendered scene is low, so the rendered scene has problems such as low accuracy and poor quality.
[0038] In order to solve the above problems, in the point cloud construction method, device, electronic device and storage medium provided in the embodiment of the present application, at least two image frames and camera parameters of the camera that collects at least two image frames are obtained; based on the at least two image frames and the camera parameters, the first camera posture information is estimated, and the first camera posture information includes the camera posture information corresponding to each image frame; based on the first camera posture information, at least one group of binocular image pairs that meet the binocular constraint conditions are determined from at least two image frames; stereo matching is performed on at least one group of binocular image pairs to construct a dense point cloud corresponding to at least two image frames. In this scheme, since the first camera posture information is used to characterize the acquisition trajectory of the camera collecting multiple image frames, based on the first camera posture information, an image pair that approximates an ideal binocular image pair can be selected from the multiple image frames. Then, the selected binocular image pair is used to reconstruct a three-dimensional dense point cloud, so that a dense point cloud with a larger number of point clouds can be constructed, so that when the electronic device performs three-dimensional scene rendering based on the dense point cloud, the geometric description accuracy and quality of the rendered scene can be improved, and the rendering effect of the three-dimensional scene is improved.
[0039] The execution subject of the point cloud construction method provided in the embodiment of the present application may be a point cloud construction device, and the point cloud construction device may be an electronic device, or a functional module in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.
[0040] The present application embodiment provides a point cloud construction method. Figure 1 FIG. 1 is a flow chart of a point cloud construction method provided in an embodiment of the present application. Figure 1 As shown, the point cloud construction method provided in the embodiment of the present application may include the following steps 101 to 104.
[0041] Step 101: The electronic device obtains at least two image frames and camera parameters of a camera that captures the at least two image frames.
[0042] In some embodiments of the present application, the at least two image frames may be ordered or unordered, which is not limited in the embodiments of the present application.
[0043] In some embodiments of the present application, the at least two image frames mentioned above may be all image frames included in the image frame sequence.
[0044] In some embodiments of the present application, the above-mentioned camera parameters are the camera parameters of the camera used by the above-mentioned electronic device when shooting the above-mentioned at least two image frames. Among them, the above-mentioned camera parameters include camera intrinsic parameters and camera extrinsic parameters. Camera intrinsic parameters are used to describe the inherent properties of the camera itself, including parameters such as focal length and pixel spacing, which determine the shape and size of the two-dimensional image obtained by the camera from the three-dimensional scene. Camera extrinsic parameters describe the position and orientation of the camera in three-dimensional space, and determine the camera's observation angle of the scene.
[0045] In some embodiments of the present application, the electronic device may use a camera module to collect a period of time-series continuous video data and calibrate the parameters of the camera module. For example, the above camera calibration may adopt a traditional camera calibration method, a camera self-calibration method, or a calibration method based on active vision, which is not limited in the embodiments of the present application.
[0046] Step 102: The electronic device estimates first camera posture information based on at least two image frames and camera parameters.
[0047] In some embodiments of the present application, the first camera posture information includes camera posture information corresponding to each image frame of the at least two image frames.
[0048] In some embodiments of the present application, the first camera posture information is used to characterize the motion trajectory of the camera when the electronic device captures the at least two image frames.
[0049] In some embodiments of the present application, the camera posture information corresponding to each of the at least two image frames includes the orientation and center point coordinates of the camera when the corresponding image frame is captured.
[0050] In some embodiments of the present application, Figure 1 ,like Figure 2 As shown, the above step 102 can be implemented through the following steps 201 to 203.
[0051] Step 201: The electronic device extracts feature points to be matched in each of at least two image frames.
[0052] In the embodiment of the present application, the feature points to be matched are points that are used to describe the image frame and have distinct features and effectively reflect the objects in the image frame.
[0053] In the embodiment of the present application, the above-mentioned feature point to be matched consists of two parts: a key point and a descriptor. Among them, the key point refers to the position, orientation, size and other information of the feature point to be matched in the image frame. The descriptor is usually a vector that describes the information of the pixels around the key point.
[0054] In some embodiments of the present application, the feature points to be matched may be contour points, bright spots in a darker area, or dark spots in a brighter area.
[0055] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 201 can be implemented by the following steps 201a and 201b.
[0056] Step 201a: The electronic device divides the first image frame into a plurality of grid images.
[0057] In some embodiments of the present application, the first image frame is one of at least two image frames. For example, the first image frame may be one of the image frames in the image frame sequence.
[0058] Step 201b: The electronic device extracts at least one first feature point from each grid image and extracts at least one second feature point from the first image frame, and uses the extracted first feature point and second feature point as feature points to be matched in the first image frame.
[0059] In some embodiments of the present application, the electronic device can extract the same number of first feature points from each grid image. Alternatively, the electronic device can extract different numbers of first feature points from each grid image. Alternatively, the electronic device can extract the same number of first feature points from a portion of the grid images, and extract different numbers of first feature points from another portion of the grid images. The specific number is determined according to actual usage requirements and is not limited by the embodiments of the present application.
[0060] It should be noted that using the extracted first feature points and second feature points as the feature points to be matched in the first image frame means: using all the first feature points extracted from all grid images and all the second feature points extracted from the first image frame as the feature points to be matched in the first image frame.
[0061] In some embodiments of the present application, the extraction of the same number of first feature points from each grid image can be implemented as follows: taking the grid as the extraction unit, the electronic device extracts the same number of first feature points from the corresponding grid images based on the importance of the feature points in the grid image. For example, assuming that the total number of feature points to be matched obtained from the first image frame is set to 900 based on the requirements of the actual scene for rendering effect and accuracy, the first image frame can be divided into 9 grid images, and the grid is taken as the extraction unit. In each grid image, the feature points are sorted according to their importance, and 50 first feature points are selected from the corresponding grid image based on the sorting result.
[0062] In some embodiments of the present application, the above-mentioned extraction of at least one second feature point from the first image frame can be implemented as follows: taking the entire first image frame as the extraction unit, in the first image frame, extracting multiple second feature points from the entire image based on the importance of the feature points in the first image frame. For example, after obtaining 450 first feature points through the above process, further, the entire first image frame can be continued to be used as the extraction unit, all feature points in the first image frame can be sorted according to the importance of the feature points, and the remaining 450 second feature points can be selected from all feature points of the first image frame according to the sorting result. Optionally, when the selected second feature point is repeated with the first feature point, the current second feature point can be eliminated and progressively filled according to the sorting result.
[0063] It should be noted that the above scenario is only an exemplary description, and the protection scope of the embodiments of the present application is not limited thereto. For example, the number of feature points to be matched, and the proportion of the first feature point and the second feature point in the feature points to be matched may also be other values according to the actual scenario.
[0064] In the embodiment of the present application, in the above-mentioned process of extracting the feature points to be matched, the first image frame is gridded and at least one first feature point is extracted from each grid image. In addition to extracting feature points in units of grids, the embodiment of the present application also extracts all feature points in the first image frame in units of the first image frame to obtain second feature points. By merging the first feature point and the second feature point together to obtain the feature point to be matched, the uniformity of the distribution of the feature points to be matched can be improved while ensuring the uniqueness of the feature point to be matched, which is beneficial to improve the quality of the point cloud constructed subsequently.
[0065] It should be noted that in the front-end algorithm of visual SLAM, extracting feature points and matching them is extremely important for back-end posture estimation and optimization. The uniform distribution of feature points also has a significant impact on the effect of the 3DGS algorithm. Therefore, in the original visual SLAM system, the embodiment of the present application grids the image during the feature point extraction process, extracts feature points for each sub-grid image, and sorts the scores of the feature points in the whole picture to ensure that there are feature points in each sub-grid image. This step ensures the uniqueness of the feature points, and also achieves a uniform distribution of feature points in three-dimensional space by evenly distributing the feature points in the image space.
[0066] In the feature point matching process of visual SLAM, the embodiment of the present application distinguishes correct feature points from outliers through the RANSAC algorithm. In this step, by setting the correct outlier ratio threshold, it is possible to identify whether there are dynamic objects in the image. Since the 3DGS algorithm cannot correctly process dynamic objects in the scene, the embodiment of the present application further marks the key frames with dynamic objects and the dynamic object parts of the image, and uses the mask image to remove them in the subsequent 3DGS algorithm, thereby improving the effect of 3DGS in dynamic scenes.
[0067] In some embodiments of the present application, after "extracting at least one first feature point from each grid image and extracting at least one second feature point from the first image frame" in the above step 201b, the point cloud construction method provided in the embodiment of the present application may include the following steps 201b1 to 201b3. In addition, "using the extracted first feature point and second feature point as the feature points to be matched in the first image frame" in the above step 201b may be implemented by the following step 301b4.
[0068] Step 201b1: the electronic device generates a first mask image based on the position information of the first feature point in the first image frame, the position information of the second feature point in the first image frame, and the first image frame.
[0069] Step 201b2: the electronic device determines a target area from the first image frame based on the first mask image.
[0070] In some embodiments of the present application, the first feature point and the second feature point do not exist in the target area.
[0071] Step 201b3: The electronic device extracts a third additional feature point from the target area.
[0072] It can be understood that the electronic device can extract supplementary feature points, ie, third feature points, from the target area.
[0073] Step 201b4: the electronic device uses the extracted first feature point, second feature point and third feature point as feature points to be matched in the first image frame.
[0074] In the embodiment of the present application, a first mask image is generated based on the position information of the first feature point in the first image frame, the position information of the second feature point in the first image frame, and the first image frame. The first mask image can be used to determine the target area in the first image frame where the first feature point and the second feature point are not extracted, so that supplementary feature points can be extracted in the target area, so that the first feature point, the second feature point, and the supplementary feature point can be used as the feature points to be matched in the first image frame, further increasing the number of feature points to be matched extracted, and thus improving the quality of the point cloud constructed subsequently.
[0075] In some embodiments of the present application, the electronic device can estimate the current posture p of the camera by minimizing the reprojection error between the three-dimensional feature point map and the two-dimensional feature points. t Then, the known feature point matching relationship is combined with the estimated camera posture p t Perform triangulation calculation on the same feature point to obtain the coordinate position information p of the feature point in the map coordinate system i .
[0076] In some embodiments of the present application, the extraction of the first feature point, the second feature point and the third feature point in the above step 201b can be implemented based on Oriented Fast and Rotated Brief (ORB) or Scale Invariant Feature Transform (SIFT).
[0077] Among them, the above-mentioned ORB uses the feature detection algorithm based on accelerated segment test (Features from accelerated segment test, FAST) algorithm to extract the key points of the above-mentioned feature points to be matched. The implementation principle is as follows: If a certain pixel point has a certain attribute difference with enough continuous pixel points in its surrounding area, and the difference is greater than a preset threshold, it can be concluded that the pixel point has a recognizable difference from its neighboring pixels, and the pixel point is determined as a key point. After determining the key point, the above-mentioned ORB also needs to generate a descriptor corresponding to the key point based on the feature point description algorithm (Binary Robust Independent Elementary Feature, BRIEF). The descriptor is a binary descriptor composed of 0 and 1, which is used to describe the surrounding image area of the key point.
[0078] The above-mentioned SIFT extracts the above-mentioned feature points to be matched by executing the following method: searching image positions in all scale spaces, identifying potential interest points with scale and rotation invariance through Gaussian differential functions; at each candidate interest point position, accurately determining the position and scale of the key point through a finely fitted model; assigning one or more directions to each key point position based on the local gradient direction of the image, and subsequent operations on the image data are transformed relative to the direction, scale and position of the key point; in the neighborhood around each key point, measuring the local gradient of the image at a selected scale, and establishing a descriptor for each key point based on the selected gradient, so as to describe the key point through a set of vectors so that it does not change with various changes, such as changes in illumination, changes in viewing angle, etc.
[0079] It should be noted that the above scenario is only an exemplary description, and the protection scope of the embodiments of the present application is not limited thereto. For example, the above feature points to be matched can also be extracted based on other feature point extraction methods.
[0080] Step 202: The electronic device performs feature point matching on the feature points to be matched in all adjacent image frames in at least two image frames to obtain at least one pair of first matching feature point pairs.
[0081] In some embodiments of the present application, the electronic device may search adjacent image frames in the above-mentioned image frame sequence based on a sequential search matching algorithm, and match the feature points to be matched in the adjacent image frames to obtain at least one pair of first matching feature points.
[0082] In the embodiment of the present application, since the image frames in the above image frame sequence are time-sequential, the feature points to be matched in adjacent image frames can be matched by a sequential search matching algorithm. Compared with the exhaustive search matching algorithm, in the process of matching the feature points to be matched, the algorithm time is greatly compressed and the time consumption is reduced.
[0083] Step 203: The electronic device estimates first camera posture information based on at least one pair of first matching feature points and camera parameters.
[0084] In some embodiments of the present application, in the above process, by extracting feature points to be matched and performing feature point matching, a first matching feature point pair is obtained, and the first camera posture information is estimated based on the first matching feature point pair and the camera parameters, thereby avoiding the use of incremental SfM to solve the camera posture in the subsequent point cloud construction process, thereby greatly reducing the time cost.
[0085] In some embodiments of the present application, Figure 2 ,like Figure 4 As shown, the above step 203 can be implemented by the following steps 203a to 203c.
[0086] Step 203a: The electronic device divides at least one pair of first matching feature point pairs into at least one matching feature point pair group.
[0087] In the embodiment of the present application, each matching feature point pair group in the at least one matching feature point pair group includes at least four pairs of first matching feature point pairs.
[0088] Step 203b: For each matching feature point pair group, the electronic device calculates the homography matrix corresponding to the matching feature point pair group, and divides at least one first matching feature point pair based on the homography matrix, dividing the first matching feature point pair that satisfies the homography matrix into an inner point, and dividing the first matching feature point pair that does not satisfy the homography matrix into an outer point.
[0089] Step 203c: The electronic device estimates first camera pose information based on the interior points in the target matching feature point pair group and the camera parameters.
[0090] In the embodiment of the present application, the target matching feature point pair group is the group with the largest number of inliers in at least one matching feature point pair group.
[0091] In some embodiments of the present application, after the above step 203c, the point cloud construction method provided in the embodiments of the present application may further include the following step 203d.
[0092] Step 203d: The electronic device marks the second image frame in which the external point in the target matching feature point pair exists, and the target object corresponding to the external point in the second image frame, among the at least two image frames.
[0093] In the embodiment of the present application, in the above process, by dividing at least one pair of first matching feature point pairs into at least one matching feature point pair group, and determining the group with the largest number of internal points in at least one matching feature point pair group as the target matching feature point pair group, the accuracy of the estimated first camera posture information can be improved. In addition, by marking the second image frame in which the external points exist and the target object corresponding to the external points, the marked target object can be extracted in the subsequent three-dimensional rendering process, thereby improving the rendering effect of the three-dimensional rendering. Moreover, since the 3DGS algorithm cannot correctly handle dynamic objects in the scene, when the marked target object is a dynamic object, the effect of the 3DGS algorithm in dynamic scenes can also be improved.
[0094] Step 103: The electronic device determines at least one group of binocular image pairs satisfying binocular constraint conditions from at least two image frames based on the first camera posture information.
[0095] In some embodiments of the present application, the above binocular constraints include the following three:
[0096] 1) There are enough matching pixels between the two image frames in the binocular image pair; that is, there is enough common coverage area between the two key frames;
[0097] 2) The angle between the z-axis of the camera postures corresponding to the two image frames in the binocular image pair is the smallest;
[0098] 3) The deviation of the camera postures corresponding to the two image frames in the binocular image pair is as small as possible (less than a threshold) except for the x and y axes.
[0099] In some embodiments of the present application, before the above step 103, the point cloud construction method provided in the embodiment of the present application can also be implemented by the following steps 301 to 303. And the above step 103 can be implemented by the following steps 304 to 305:
[0100] Step 301: The electronic device extracts feature points to be matched in each of at least two image frames.
[0101] Step 302: The electronic device determines the reprojection error of the feature point to be matched in the corresponding image frame based on the projection coordinates of each feature point to be matched in the corresponding image frame.
[0102] Step 303: The electronic device determines an error function corresponding to the first camera posture information based on the reprojection errors corresponding to all feature points to be matched.
[0103] Step 304: The electronic device adjusts the first camera posture information based on the error function.
[0104] Step 305: The electronic device determines at least one group of binocular image pairs satisfying binocular constraint conditions from at least two image frames based on the adjusted first camera posture information.
[0105] In the embodiment of the present application, since the estimated value of the first camera posture information is obtained in the above steps, in the embodiment of the present application, the first camera posture information can be optimized by the one-time global adjustment optimization method of the above steps 301 to 304. Compared with the incremental adjustment optimization, the algorithm time is further compressed while ensuring the optimization accuracy.
[0106] In some embodiments of the present application, in combination with specific scenarios, the process of adjusting the first camera posture information based on the global adjustment optimization algorithm in steps 301 to 304 is described in detail:
[0107] In this scenario, the camera posture information p corresponding to each image frame is i Defined on the special Euclidean group SE(3), for the feature point p to be matched j ∈R 3 , whose three-dimensional coordinates in the world coordinate system are defined as In the image I i The corresponding two-dimensional coordinate in the camera coordinate system is defined as u i,j , and can be obtained through the following projection function π:R 3 →R 2 Determine the feature point to be matched in image I i The reprojected coordinates in:
[0108] u′ i,j =π(R i p j +t i ,K i )
[0109] in, constitutes the camera posture at time i, K i is the camera internal parameter at time i. The corresponding residual is defined as:
[0110] r i,j =u i,j -π(R i p j +t i ,K i )
[0111] By summing up the observation residuals of all feature points to be matched under all camera postures, the overall target residual of the adjustment problem can be obtained:
[0112]
[0113] Among them, n c is the number of camera pose information corresponding to the image frame, n p is the number of feature points to be matched. Minimizing the residual error can be equivalent to minimizing the linearization error of the perturbation amount as follows:
[0114]
[0115] Among them, J is the corresponding linearized Jacobian matrix, and the above formula is converted into the normal equation form as follows:
[0116] J T J(-Δx)=J T
[0117] By solving the above normal equations, the state quantity x can be iterated to obtain the optimal solution.
[0118] Preferably, in the process of solving the normal equations, if the conventional method is used to simultaneously solve the camera attitude increment Δx p and feature point coordinate increment Δx l , the amount of calculation is large and the algorithm optimization time is long. In order to solve this problem, the embodiment of the present application can also decompose the problem and solve the camera posture increment Δx separately p and feature point coordinate increment Δx l , which can improve the speed of the algorithm while ensuring the accuracy of the algorithm. The specific implementation is as follows:
[0119] Let H = J T J can be decomposed as follows:
[0120]
[0121] in, Arranging the normal equations according to the decomposed H matrix, we can obtain:
[0122]
[0123] in,
[0124] Performing Shure Complement on the above formula, we can calculate the camera attitude increment Δx p and feature point coordinate increment Δx l .
[0125] In some embodiments of the present application, since the camera posture corresponding to the binocular image pair determined by the above process constitutes an approximate binocular, after the above step 305 "based on the adjusted first camera posture information, at least one group of binocular image pairs satisfying the binocular constraint condition is determined from at least two image frames", the embodiment of the present application can also correct the acquired binocular image pair based on the binocular correction algorithm through the following steps 306 and 307:
[0126] Step 331: The electronic device determines the relative position and posture between the cameras at the corresponding moments of the two image frames based on the acquired camera posture information corresponding to the two image frames in the binocular image pair.
[0127] Step 332: The electronic device deforms the binocular image pair through a binocular correction algorithm based on the camera intrinsic parameters and relative and relative positions at the corresponding moment to obtain a corrected horizontal binocular image or a vertical binocular image.
[0128] Step 104: The electronic device performs stereo matching on at least one set of binocular image pairs to construct a dense point cloud corresponding to at least two image frames.
[0129] In some embodiments of the present application, after obtaining the above binocular image pair, the above step 104 performs stereo matching on at least one group of binocular image pairs, and the process of constructing a dense point cloud corresponding to at least two image frames can be implemented by the following steps 340a and 340b:
[0130] Step 104a: The electronic device obtains the correspondence between the pixels in the two image frames of the binocular image pair and performs pixel matching.
[0131] Step 104b: The electronic device constructs a dense point cloud corresponding to at least two image frames based on the matching relationship between the pixel points and the relative posture between the cameras corresponding to the two image frames.
[0132] In the embodiment of the present application, since the first camera posture information is used to characterize the acquisition trajectory of the camera in acquiring the multiple image frames, an approximately ideal binocular image pair can be selected from the multiple image frames based on the first camera posture information. Then, the selected binocular image pair is used to reconstruct a three-dimensional dense point cloud, so that a dense point cloud with a larger number of point clouds can be constructed, so that when the electronic device performs three-dimensional scene rendering based on the dense point cloud, the geometric description accuracy of the rendered scene can be improved, thereby improving the rendering effect of the three-dimensional scene.
[0133] In some embodiments of the present application, after the above step 104, the point cloud construction method provided in the embodiment of the present application further includes the following step 401.
[0134] Step 401: The electronic device renders at least two image frames based on a dense point cloud.
[0135] In some embodiments of the present application, the rendering content in the second image frame does not include the target object.
[0136] It can be understood that, in the process of rendering at least two image frames based on the dense point cloud, the electronic device removes the target object in the second image frame, so that the rendered content in the second image frame does not include the target object.
[0137] In some embodiments of the present application, after the above step 102, the point cloud construction method provided by the embodiment of the present application further includes the following step 402. And after the above step 104, the point cloud construction method provided by the embodiment of the present application may further include the following step 403.
[0138] Step 402: The electronic device constructs a sparse point cloud corresponding to at least two image frames based on at least two image frames and first camera posture information.
[0139] Step 403: The electronic device fuses the sparse point cloud and the dense point cloud to obtain a fused point cloud.
[0140] In an embodiment of the present application, by fusing sparse point clouds and dense point clouds, the number of point clouds constructed by the dense point cloud can be further increased, so that when the electronic device performs three-dimensional scene rendering based on the dense point cloud, it has a higher-precision geometric description of the rendered scene, further improving the rendering effect of the three-dimensional scene.
[0141] An embodiment of the present application provides a point cloud construction method. Since the first camera posture information is used to characterize the acquisition trajectory of the camera to acquire multiple image frames, an approximately ideal binocular image pair can be selected from the multiple image frames based on the first camera posture information. Then, the selected binocular image pair is used to reconstruct a three-dimensional dense point cloud, so that a dense point cloud with a larger number of point clouds can be constructed, so that when the electronic device performs three-dimensional scene rendering based on the dense point cloud, the geometric description accuracy and quality of the rendered scene can be improved, thereby improving the rendering effect of the three-dimensional scene.
[0142] The overall process of the above point cloud construction is described in detail below in conjunction with a specific application scenario. In this embodiment, the input data of the point cloud construction method is collected by an electronic device, and the input data in this application scenario is time-series continuous video data. Figure 5 As shown, this embodiment includes the following steps 410 to 460.
[0143] Step 410: The electronic device estimates camera posture information corresponding to the video data based on visual SLAM.
[0144] In an embodiment of the present application, the camera posture information of the camera is estimated based on visual SLAM when the above video data is collected through a mobile phone terminal.
[0145] In an embodiment of the present application, the above-mentioned camera posture information includes camera posture information corresponding to each image frame in the video data, for example, the direction and center point coordinates of the camera when shooting the corresponding image frame.
[0146] In the embodiment of the present application, the input of visual SLAM is video data and camera parameters captured by an electronic device, and the output is camera posture information and sparse point cloud. Specifically, the above visual SLAM process includes the following steps 510 to 540:
[0147] Step 510: The electronic device extracts feature points to be matched for each image feature in the video data.
[0148] In the embodiment of the present application, the above extraction of the feature points to be matched of each image feature in the video data can be implemented as follows: for each image frame in the video data, the image frame is divided into multiple grid images, and at least one first feature point is extracted from each grid image with the grid as the extraction unit; with the image frame as the extraction unit, multiple second feature points are extracted from the entire image frame; the first feature point and the second feature point are used as the feature points to be matched in the image frame. The specific extraction process is described in detail in the above point cloud construction method, and will not be repeated here.
[0149] In the feature point extraction process, the embodiment of the present application grids the image frame, extracts the feature points of each grid image to obtain the first feature point; sorts the feature points in the image frame and extracts the second feature point, which ensures the uniformity of the feature points to be matched and the uniqueness of the feature points to be matched. In addition, by ensuring that the feature points to be matched are evenly distributed in the image space, the uniform distribution of the feature points to be matched in the three-dimensional space is achieved.
[0150] Step 520: The electronic device matches the feature points to be matched.
[0151] In the embodiment of the present application, since each image frame in the input video data has the characteristic of temporal continuity, a sequential search matching algorithm can be used instead of an exhaustive search matching algorithm to sequentially match the feature points to be matched between adjacent image frames. The specific matching process is described in detail in the above point cloud construction method and will not be repeated here.
[0152] In addition, in the matching process of the feature points to be matched in the above-mentioned visual SLAM, the embodiment of the present application can also distinguish the correct matching points and outlier matching points in the above-mentioned matching process through the RANSAC algorithm. In the embodiment of the present application, by setting the correct outlier ratio threshold in the RANSAC algorithm, it is possible to identify whether there are dynamic objects in the image. Since the 3DGS algorithm cannot correctly handle dynamic objects in the scene, after identifying the dynamic objects, the embodiment of the present application can mark the image frames with dynamic objects and the dynamic objects in the image frames, so that the mask image can be used in the subsequent 3DGS algorithm to eliminate outlier matching points, thereby improving the effect of 3DGS in dynamic scenes.
[0153] Step 530: The electronic device estimates camera posture information.
[0154] In the embodiment of the present application, based on the visual SLAM algorithm, the camera posture information is estimated by minimizing the reprojection error of the three-dimensional feature point map and the two-dimensional feature points. The specific implementation can be as follows: based on the geometric relationship between the camera planes corresponding to the adjacent image frames corresponding to the above-mentioned matching feature point pairs, the feature space points corresponding to the matching feature point pairs in the real space are determined; the feature space points are reprojected to the camera planes corresponding to the adjacent image frames to obtain the reprojection coordinates of the feature space points in the camera plane; the pixel coordinates of the matching feature point pairs in the camera plane and the difference between the reprojection coordinates are calculated to obtain the reprojection error; the camera posture information is estimated by minimizing the reprojection error.
[0155] Step 540: The electronic device constructs a sparse point cloud.
[0156] In an embodiment of the present application, this step is based on a visual SLAM algorithm, which utilizes the known matching relationship of feature points to be matched and combines the estimated camera posture information to perform triangulation calculations on the feature points to be matched, obtains the coordinates of the feature points to be matched in the world coordinate system, and constructs a sparse point cloud.
[0157] Step 420: For areas where the visual SLAM does not extract feature points to be matched in each image frame, the electronic device extracts supplementary feature points in the corresponding image frame and performs matching.
[0158] In the embodiment of the present application, for the area where the feature points to be matched are not extracted by visual SLAM in step 410, additional feature points are extracted. This can be achieved as follows: obtaining the extraction results of the feature points to be matched in step 510, and generating a mask image based on the extraction results; and extracting feature points again in the area without feature points based on the mask image. By extracting additional feature points, the number of feature points to be matched is further increased, which is conducive to improving the accuracy of subsequent adjustment optimization.
[0159] Since the image frames in the video data input through SLAM are continuous in time sequence, in the embodiment of the present application, the supplementary feature points in adjacent image frames can also be matched through a sequential search matching algorithm.
[0160] Step 430: The electronic device removes outliers from the above feature point matching results.
[0161] In an embodiment of the present application, after obtaining the matching relationship between adjacent image frames through steps 520 and 420, outliers are removed by using the basic matrix as a constraint in combination with the RANSAC algorithm, and matching feature points that do not meet the baseline constraints or have large baseline projection errors are removed to ensure the correctness of the input data in the process of triangulating feature points.
[0162] Step 440: The electronic device performs triangulation operations on the feature points and constructs a sparse point cloud.
[0163] In the embodiment of the present application, the matching relationship between the known feature points to be matched and the same feature point p i In each image frame I t , I_ (t+1) and I_ (t+k) The pixel position is minimized through the optimization algorithm to minimize the reprojection error, obtain the three-dimensional coordinates of the feature points in the map coordinate system, and construct a sparse point cloud.
[0164] Step 450: The electronic device performs global adjustment optimization on the camera posture information and the three-dimensional coordinates of the feature points.
[0165] In the embodiment of the present application, the global adjustment optimization process of step 450 includes the following S1 to S4:
[0166] S1. The electronic device performs parameter initialization and residual definition based on the camera posture information and the three-dimensional coordinates of the feature points.
[0167] In the embodiment of the present application, the camera posture information p corresponding to each image frame is i Defined on the special Euclidean group SE(3), for the feature point p to be matched j ∈R 3 , whose three-dimensional coordinates in the world coordinate system are defined as In the image I i The corresponding two-dimensional coordinate in the camera coordinate system is defined as u i,j , and can be obtained through the following projection function π:R 3 →R 2 Determine the feature point to be matched in image I i The reprojected coordinates in:
[0168] u′ i,j =π(Ri p j +t i ,K i )
[0169] in, constitutes the camera posture at time i, K i is the camera internal parameter at time i. The corresponding residual is defined as:
[0170] r i,j =u i,j -π(R i p j +t i ,K i )
[0171] S2, electronic equipment obtains the overall target residual of the adjustment problem and converts the formula into a normal equation.
[0172] By summarizing the observation residuals of all feature points to be matched under all camera postures, the overall target residual of the adjustment problem can be obtained as follows:
[0173]
[0174] Among them, n c is the number of camera pose information corresponding to the image frame, n p is the number of feature points to be matched.
[0175] Minimizing the residual can be equivalent to minimizing the linearization error of the disturbance as follows:
[0176]
[0177] Among them, J is the corresponding linearized Jacobian matrix, and the above formula is converted into the normal equation form as follows:
[0178] J T J(-Δx)=J T
[0179] S3. The electronic device solves the above normal equations.
[0180] S4. The electronic device solves the above normal equations and iteratively updates the state quantity x to obtain the optimal solution.
[0181] Optionally, in the process of solving the normal equations, if the conventional method is used to simultaneously solve the camera attitude increment Δx p and feature point coordinate increment Δx l , the amount of calculation is large and the algorithm optimization time is long. In order to solve this problem, the embodiment of the present application can also decompose the problem and solve the camera posture increment Δx separately pand feature point coordinate increment Δx l , which can improve the speed of the algorithm while ensuring the accuracy of the algorithm. The specific implementation is as follows:
[0182] Let H = J T J can be decomposed as follows:
[0183]
[0184] in, Arranging the normal equations according to the decomposed H matrix, we can obtain:
[0185]
[0186] in,
[0187] Performing Shure Complement on the above formula, we can calculate the camera attitude increment Δx p and feature point coordinate increment Δx l .
[0188] In the embodiment of the present application, since the estimated value of the first camera attitude information is obtained in the above steps, the first camera attitude information can be optimized by a one-time global adjustment optimization method. Compared with the incremental adjustment optimization, while ensuring the optimization accuracy, the algorithm time is further compressed. And by separately solving the camera attitude increment Δx p and feature point coordinate increment Δx l , further improving the algorithm speed and accuracy.
[0189] Step 460: The electronic device performs binocular dense reconstruction based on the optimized camera posture information.
[0190] In the embodiment of the present application, after obtaining high-precision camera posture information through the above-mentioned global adjustment optimization, it is necessary to determine the image frames for binocular reconstruction from the video data in combination with the basic constraints of the ideal binocular. Specifically, the binocular image pairs for binocular dense reconstruction can be selected based on the following conditions:
[0191] 1) There is enough common coverage area between the two image frames;
[0192] 2) The angle between the z-axes of the camera coordinate systems of the camera poses corresponding to the two image frames is the smallest;
[0193] 3) The deviation of the camera coordinate system of the camera postures corresponding to the two image frames is as small as possible except for the x and y axes.
[0194] Specifically, the selection process can be implemented as follows: combining the posture graph of the camera posture information and the matching relationship between the feature points to be matched, and traversing the image frame combination in the motion trajectory of the camera to collect video data, find the camera pair that is most suitable for binocular dense reconstruction.
[0195] It should be noted that the relationship between the binocular image pairs determined by the above process is approximate binocular. In order to make the above binocular image pairs more consistent with the binocular camera, after determining the relative pose between the cameras at the corresponding moments of the two image frames based on the camera pose information corresponding to the two image frames in the acquired binocular image pair, and the camera intrinsic parameters at the corresponding moments of the two image frames, the binocular correction algorithm can also be used to obtain the correction matrix and projection matrix under the two poses, and the binocular image pair is deformed based on the correction matrix and the projection matrix to obtain a corrected horizontal binocular image pair or a vertical binocular image pair. For example, assuming that the camera at time i and the camera at time j constitute an approximate binocular, after obtaining the relative pose between the cameras at time i and time j, and the camera intrinsic parameters at the two moments through adjustment optimization. The correction matrix R under the two poses can be obtained by the binocular correction algorithm. i ,R j And the projection matrix P i ,P j By warping the image, a horizontal binocular image or a vertical binocular image that meets the requirements can be obtained.
[0196] After obtaining the corrected binocular image pair, the correspondence between the pixels of the two image frames can be obtained through the binocular stereo matching and reconstruction algorithm. When the relative posture between the cameras is known, the three-dimensional reconstruction of the dense point cloud can be completed based on the matching relationship between the pixels and the relative posture between the cameras.
[0197] Furthermore, after constructing dense point clouds corresponding to at least two image frames, since the scale relationship between cameras is consistent after adjustment optimization, it is also possible to obtain a fused point cloud by fusing the dense point cloud reconstructed by binocular and the sparse point cloud after global adjustment optimization, and initialize three-dimensional rendering based on the fused point cloud.
[0198] The point cloud construction method provided in the embodiment of the present application has the following advantages:
[0199] (1) Shortened algorithm time. The embodiment of the present application significantly reduces the number of feature point extractions, and combines the temporal relationship of the input image to replace the feature point matching method, which greatly compresses the algorithm time in the step of building data association. In the process of optimizing the camera posture information and three-dimensional point coordinates, a one-time global adjustment optimization replaces the incremental adjustment optimization, which further compresses the algorithm time while ensuring the optimization accuracy.
[0200] (2) The constructed point cloud is denser. After global adjustment optimization, the embodiment of the present application traverses the camera posture, selects binocular image pairs that are close to the ideal binocular, and performs binocular correction and three-dimensional dense reconstruction based on the binocular image pairs, which greatly increases the number of point clouds and makes the point cloud more accurate in describing the geometric shape of the scene.
[0201] (3) Better three-dimensional rendering effect. The point cloud constructed by the embodiment of the present application is used to initialize the subsequent 3DGS rendering. Since the point cloud constructed by the embodiment of the present application is denser, the number of iterations of three-dimensional Gaussian splitting is reduced in the subsequent 3DGS training, and a more complete description of the scene is provided, the quality of the rendered video is better, and the training speed is faster.
[0202] It should be noted that the execution subject of the point cloud construction method provided in the embodiment of the present application may be a point cloud construction device, or an electronic device, or may also be a functional module or entity in an electronic device. In the embodiment of the present application, the execution of the point cloud construction method by a point cloud construction device is taken as an example to illustrate the point cloud construction device provided in the embodiment of the present application.
[0203] Figure 6 A possible structural diagram of a point cloud construction device involved in an embodiment of the present application is shown. Figure 6 As shown, the point cloud construction device 600 may include: an acquisition module 610, a processing module 620 and a determination module 630. Among them:
[0204] The acquisition module 610 is used to acquire at least two image frames and camera parameters of a camera that captures the at least two image frames.
[0205] The processing module 620 is used to estimate first camera posture information based on at least two image frames and camera parameters acquired by the acquisition module 610, where the first camera posture information includes camera posture information corresponding to each image frame.
[0206] The determination module 630 is used to determine at least one group of binocular image pairs satisfying the binocular constraint condition from at least two image frames based on the first camera posture information obtained by the processing module 620.
[0207] The processing module 620 is further configured to perform stereo matching on at least one set of binocular image pairs determined by the determination module 630 to construct a dense point cloud corresponding to at least two image frames.
[0208] In one possible implementation, the processing module 620 is specifically used to: extract feature points to be matched in each of at least two image frames; perform feature point matching on feature points to be matched in all adjacent image frames in at least two image frames to obtain at least one pair of first matching feature point pairs; and estimate first camera posture information based on at least one pair of first matching feature point pairs and camera parameters.
[0209] In a possible implementation, the processing module 620 is specifically used to: divide the first image frame into multiple grid images, where the first image frame is one of at least two image frames; extract at least one first feature point from each grid image, and extract at least one second feature point from the first image frame, and use the extracted first feature points and second feature points as feature points to be matched in the first image frame.
[0210] In a possible implementation, the processing module 620 is further used to generate a first mask image based on the position information of the first feature point in the first image frame, the position information of the second feature point in the first image frame, and the first image frame after extracting at least one first feature point from each grid image and extracting at least one second feature point from the first image frame; based on the first mask image, determine the target area from the first image frame, where the first feature point and the second feature point do not exist; and extract a third feature point from the target area. The processing module 620 is specifically used to use the extracted first feature point, the second feature point, and the third feature point as the feature points to be matched in the first image frame.
[0211] In a possible implementation, the processing module 620 is specifically used to: divide at least one pair of first matching feature point pairs into at least one matching feature point pair group, each matching feature point pair group containing at least four pairs of first matching feature point pairs; for each matching feature point pair group, calculate the homography matrix corresponding to the matching feature point pair group, and divide at least one first matching feature point pair based on the homography matrix, divide the first matching feature point pairs that satisfy the homography matrix into internal points, and divide the first matching feature point pairs that do not satisfy the homography matrix into external points; estimate the first camera posture information based on the internal points and camera parameters in the target matching feature point pair group; the target matching feature point pair group is the group with the largest number of internal points in at least one matching feature point pair group.
[0212] In a possible implementation, the processing module 620 is further used to mark the second image frame in which there are external points in the target matching feature point pair group in at least two image frames, and the target object corresponding to the external points in the second image frame. The processing module 620 is also used to perform stereo matching on at least one set of binocular image pairs and construct a dense point cloud corresponding to at least two image frames, and then render the at least two image frames based on the dense point cloud; wherein the rendered content in the second image frame does not include the target object.
[0213] In a possible implementation, the processing module 620 is further used to extract the feature points to be matched in each of the at least two image frames before the determination module 630 determines at least one set of binocular image pairs satisfying the binocular constraint condition from the at least two image frames based on the first camera posture information. The determination module 630 is further used to determine the reprojection error of the feature points to be matched in the corresponding image frame based on the projection coordinates of each feature point to be matched in the corresponding image frame; and to determine the error function corresponding to the first camera posture information based on the reprojection errors corresponding to all feature points to be matched. The determination module 630 is specifically used to adjust the first camera posture information based on the error function; and to determine at least one set of binocular image pairs satisfying the binocular constraint condition from the at least two image frames based on the adjusted first camera posture information.
[0214] In a possible implementation, the processing module 620 is further used to construct a sparse point cloud corresponding to at least two image frames based on the at least two image frames and the first camera posture information after estimating the first camera posture information based on the at least two image frames and the camera parameters. The processing module 620 is further used to fuse the sparse point cloud and the dense point cloud to obtain a fused point cloud after performing stereo matching on at least one set of binocular image pairs and constructing a dense point cloud corresponding to at least two image frames.
[0215] The point cloud construction device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0216] The point cloud construction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0217] The point cloud construction device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be repeated here.
[0218] Alternatively, if Figure 7 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned point cloud construction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0219] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0220] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0221] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.
[0222] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0223] The processor 1010 is used to obtain at least two image frames. The processor 1010 is used to obtain camera parameters of a camera that captures at least two image frames; based on the at least two image frames and the camera parameters, estimate first camera posture information, the first camera posture information including camera posture information corresponding to each image frame; based on the first camera posture information, determine at least one group of binocular image pairs that meet binocular constraint conditions from at least two image frames; perform stereo matching on at least one group of binocular image pairs to construct a dense point cloud corresponding to at least two image frames.
[0224] In the electronic device provided in the embodiment of the present application, since the first camera posture information includes the camera posture information corresponding to each image frame, the phase posture between any two image frames can be obtained through the first camera posture information, and then the binocular image pair of binocular constraint conditions can be determined from at least two image frames. Since the binocular image pair satisfies the binocular constraint conditions, the number of matchable pixel points contained between the two image frames of the binocular image pair is large and meets the posture conditions of the binocular camera, so that the number of point clouds of the dense point cloud constructed by binocular reconstruction based on the binocular image pair is also large, and the geometric description of the scene is more accurate. Furthermore, in a three-dimensional rendering scene, compared with a sparse point cloud, the use of the dense point cloud can provide a more complete scene description and better rendering effects.
[0225] Optionally, in an embodiment of the present application, the above-mentioned processor 1010 is specifically used to: extract feature points to be matched in each image frame of at least two image frames; perform feature point matching on feature points to be matched in all adjacent image frames of at least two image frames to obtain at least one pair of first matching feature point pairs; and estimate first camera posture information based on at least one pair of first matching feature point pairs and camera parameters.
[0226] Optionally, in an embodiment of the present application, the processor 1010 is specifically used to: divide the first image frame into multiple grid images, the first image frame being one of at least two image frames; extracting at least one first feature point from each grid image, and extracting at least one second feature point from the first image frame, and using the extracted first feature points and second feature points as feature points to be matched in the first image frame.
[0227] Optionally, in the embodiment of the present application, the processor 1010 is further used to generate a first mask image based on the position information of the first feature point in the first image frame, the position information of the second feature point in the first image frame, and the first image frame after extracting at least one first feature point from each grid image and extracting at least one second feature point from the first image frame; based on the first mask image, determine the target area from the first image frame, where the first feature point and the second feature point do not exist; and extract a third feature point from the target area. The processor 1010 is specifically used to use the extracted first feature point, the second feature point, and the third feature point as the feature points to be matched in the first image frame.
[0228] Optionally, in an embodiment of the present application, the above-mentioned processor is specifically used to: divide the at least one pair of first matching feature point pairs into at least one matching feature point pair group, each matching feature point pair group containing at least four pairs of first matching feature point pairs; for each matching feature point pair group, calculate the homography matrix corresponding to the matching feature point pair group, and based on the homography matrix, divide at least one first matching feature point pair, divide the first matching feature point pairs that satisfy the homography matrix into internal points, and divide the first matching feature point pairs that do not satisfy the homography matrix into external points; based on the internal points in the target matching feature point pair group and the camera parameters, estimate the first camera posture information; the target matching feature point pair group is the group with the largest number of internal points in at least one matching feature point pair group.
[0229] Optionally, in an embodiment of the present application, the processor 1010 is further used to mark the second image frame in which the external point in the target matching feature point pair group exists in the at least two image frames, and the target object corresponding to the external point in the second image frame. The processor 1010 is also used to perform stereo matching on at least one set of binocular image pairs and construct a dense point cloud corresponding to at least two image frames, and then render the at least two image frames based on the dense point cloud; wherein the rendered content in the second image frame does not include the target object.
[0230] Optionally, in an embodiment of the present application, the processor 1010 is further used to extract the feature points to be matched in each of the at least two image frames before determining at least one set of binocular image pairs satisfying the binocular constraint condition from the at least two image frames based on the first camera posture information; determine the reprojection error of the feature points to be matched in the corresponding image frame based on the projection coordinates of each feature point to be matched in the corresponding image frame; and determine the error function corresponding to the first camera posture information based on the reprojection errors corresponding to all feature points to be matched. The processor 1010 is specifically used to: adjust the first camera posture information based on the error function; and determine at least one set of binocular image pairs satisfying the binocular constraint condition from the at least two image frames based on the adjusted first camera posture information.
[0231] Optionally, in an embodiment of the present application, the processor 1010 is further configured to, after estimating the first camera pose information based on at least two image frames and camera parameters, construct a sparse point cloud corresponding to at least two image frames based on the at least two image frames and the first camera pose information. The processor 1010 is further configured to, after performing stereo matching on at least one set of binocular image pairs and constructing a dense point cloud corresponding to at least two image frames, fuse the sparse point cloud and the dense point cloud to obtain a fused point cloud.
[0232] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.
[0233] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.
[0234] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0235] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0236] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.
[0237] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0238] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0239] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0240] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0241] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned incoming call processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0242] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0243] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0244] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A point cloud construction method, characterized in that: include: Acquire at least two image frames and camera parameters of a camera that acquires the at least two image frames; Based on the at least two image frames and the camera parameters, estimating first camera pose information, the first camera pose information comprising camera pose information corresponding to each image frame; Based on the first camera posture information, determining at least one group of binocular image pairs satisfying a binocular constraint condition from the at least two image frames; Stereo matching is performed on the at least one set of binocular image pairs to construct a dense point cloud corresponding to the at least two image frames.
2. The method according to claim 1, characterized in that The estimating first camera posture information based on the at least two image frames and the camera parameters includes: Extracting feature points to be matched in each of the at least two image frames; Performing feature point matching on the feature points to be matched in all adjacent image frames of the at least two image frames to obtain at least one pair of first matching feature point pairs; The first camera pose information is estimated based on the at least one first matching feature point pair and the camera parameters.
3. The method according to claim 2, characterized in that The extracting feature points to be matched in each of the at least two image frames comprises: Dividing a first image frame into a plurality of grid images, the first image frame being one of the at least two image frames; At least one first feature point is extracted from each grid image, and at least one second feature point is extracted from the first image frame, and the extracted first feature point and the second feature point are used as feature points to be matched in the first image frame.
4. The method according to claim 3, characterized in that After extracting at least one first feature point from each grid image and extracting at least one second feature point from the first image frame, the method further includes: generating a first mask image based on the position information of the first feature point in the first image frame, the position information of the second feature point in the first image frame, and the first image frame; Based on the first mask image, determining a target area from the first image frame, wherein the first feature point and the second feature point do not exist in the target area; Supplementary extraction of a third feature point from the target area; The step of using the extracted first feature point and the second feature point as feature points to be matched in the first image frame includes: The first feature point, the second feature point and the third feature point extracted are used as feature points to be matched in the first image frame.
5. The method according to claim 2, characterized in that: The estimating the first camera posture information based on the at least one pair of first matching feature point pairs and the camera parameters includes: Dividing the at least one pair of first matching feature point pairs into at least one matching feature point pair group, each matching feature point pair group including at least four pairs of first matching feature point pairs; For each matching feature point pair group, a homography matrix corresponding to the matching feature point pair group is calculated, and based on the homography matrix, the at least one first matching feature point pair is divided, and the first matching feature point pair that satisfies the homography matrix is divided into an inner point, and the first matching feature point pair that does not satisfy the homography matrix is divided into an outer point; The first camera posture information is estimated based on the inliers in the target matching feature point pair group and the camera parameters; the target matching feature point pair group is the group with the largest number of inliers in the at least one matching feature point pair group.
6. The method according to claim 5, characterized in that The method further comprises: Marking a second image frame in which an outlier in the target matching feature point pair exists among the at least two image frames, and a target object corresponding to the outlier in the second image frame; After performing stereo matching on the at least one set of binocular image pairs and constructing a dense point cloud corresponding to the at least two image frames, the method further includes: rendering the at least two image frames based on the dense point cloud; The rendering content in the second image frame does not include the target object.
7. The method according to claim 1, characterized in that Before determining at least one group of binocular image pairs satisfying binocular constraint conditions from the at least two image frames based on the first camera posture information, the method further includes: Extracting feature points to be matched in each of the at least two image frames; Determining a reprojection error of the feature point to be matched in the corresponding image frame based on the projection coordinates of each feature point to be matched in the corresponding image frame; Determine an error function corresponding to the first camera posture information based on the reprojection errors corresponding to all feature points to be matched; The step of determining at least one group of binocular image pairs satisfying a binocular constraint condition from the at least two image frames based on the first camera posture information comprises: Based on the error function, adjusting the first camera posture information; Based on the adjusted first camera posture information, at least one group of binocular image pairs satisfying a binocular constraint condition is determined from the at least two image frames.
8. The method according to any one of claims 1 to 7, characterized in that: After estimating the first camera posture information based on the at least two image frames and the camera parameters, the method further includes: Based on the at least two image frames and the first camera posture information, construct a sparse point cloud corresponding to the at least two image frames; After performing stereo matching on the at least one set of binocular image pairs and constructing a dense point cloud corresponding to the at least two image frames, the method further includes: The sparse point cloud and the dense point cloud are fused to obtain a fused point cloud.
9. A point cloud construction device, characterized in that: include: Acquisition module, processing module and determination module; The acquisition module is used to acquire at least two image frames and camera parameters of a camera that acquires the at least two image frames; The processing module is used to estimate first camera posture information based on the at least two image frames and the camera parameters acquired by the acquisition module, where the first camera posture information includes camera posture information corresponding to each image frame; The determining module is configured to determine at least one group of binocular image pairs satisfying a binocular constraint condition from the at least two image frames based on the first camera posture information obtained by the processing module; The processing module is further used to perform stereo matching on the at least one group of binocular image pairs determined by the determination module to construct a dense point cloud corresponding to the at least two image frames.
10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the point cloud construction method according to any one of claims 1 to 8 are implemented.
11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the point cloud construction method according to any one of claims 1 to 8 are implemented.
Citation Information
Cited By
Binocular vision and Gaussian sputtering fused three-dimensional depth reconstruction method, device, equipment and medium
CN119693437A
3D Gaussian splash visual repositioning method based on visual basic model driving
CN119963641A
Temporally stable initialization for the generation of dynamic 3D gaussians from multi-view video captures
EP4776224A1
Temporally stable initialization for the generation of dynamic 3D gaussians from multi-view video captures
WO2026149919A1
