A three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The application discloses a three-dimensional reconstruction method. For each image frame in a video, sparse feature vectors and dense feature vectors of pixel points of the image frame are extracted. For each image frame, pixel point pairs matched between the image frame and a to-be-matched image frame are determined based on the sparse feature vectors of the pixel points of the image frame and the sparse feature vectors of the pixel points of the to-be-matched image frame. The dense feature vectors of each pixel point in the pixel point pairs are used to screen out pixel point pairs with a similarity higher than a first preset threshold as target pixel point pairs. According to adjacent two frames and all target pixel point pairs corresponding to the adjacent two frames, camera extrinsic parameters and initial three-dimensional points of each pixel point in the target pixel point pairs are calculated. The initial three-dimensional points are projected back to a pixel plane by using camera intrinsic parameters and the calculated extrinsic parameters to obtain projection points. A loss function is constructed, and the camera extrinsic parameters, the camera intrinsic parameters and the three-dimensional points are iteratively optimized until the best estimated values are found when the loss function reaches a minimum value.

Description

Technical Field

[0001] This application relates to the field of visual technology, specifically to a three-dimensional reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] 3D reconstruction technology is the process of recovering the 3D model of the object recorded in a series of 2D images. It is a key technology for building 3D models that represent the real world in a computer. How to achieve 3D reconstruction in a low-cost manner has been a problem that those skilled in the art have been working on. Summary of the Invention

[0003] In view of this, this application provides a three-dimensional reconstruction method, apparatus, device and storage medium that can reduce the cost of three-dimensional reconstruction.

[0004] To solve the above problems, the technical solution provided in this application is as follows:

[0005] In a first aspect of this application, a three-dimensional reconstruction method is provided, the method comprising:

[0006] For the image frames of the video to be processed, extract the sparse feature vectors and dense feature vectors of the pixels in the image frames;

[0007] For each image frame, based on the sparse feature vector of the pixels in the image frame and the sparse feature vector of the pixels in the image frame to be matched, a pixel pair consisting of the pixels of the image frame and the image frame to be matched is determined. The image frame to be matched is an image frame in the video to be processed that satisfies a preset adjacent relationship with the image frame.

[0008] Based on the similarity of the dense feature vectors of the two pixels in the pixel pair, pixel pairs with a similarity higher than a first preset threshold are selected as target pixel pairs.

[0009] Based on the two adjacent image frames of the video to be processed and all target pixel pairs corresponding to the two adjacent image frames, the initial extrinsic parameters of the camera and the initial three-dimensional point corresponding to each pixel in the target pixel pair are obtained.

[0010] Based on the initial extrinsic and initial intrinsic parameters of the camera, the initial 3D point corresponding to each pixel in the target pixel pair is projected onto the pixel plane to obtain the projection point corresponding to each pixel in the target pixel pair.

[0011] A loss function is constructed based at least on the projection points and the target pixel pair. With the goal of minimizing the loss function, the initial extrinsic parameters of the camera, the initial intrinsic parameters of the camera, and the initial 3D points are optimized until the camera extrinsic parameters, camera intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

[0012] In a second aspect of this application, a three-dimensional reconstruction apparatus is provided, the apparatus comprising:

[0013] The acquisition unit is used to extract the sparse feature vector and dense feature vector of the pixels of the image frame for the video to be processed;

[0014] The acquisition unit is further configured to, for each image frame, determine the pixel pair formed by the matching pixels of the image frame and the image frame to be matched, based on the sparse feature vector of the pixels of the image frame and the sparse feature vector of the pixels of the image frame to be matched, wherein the image frame to be matched is an image frame in the video to be processed that satisfies a preset adjacent relationship with the image frame.

[0015] The acquisition unit is further configured to select pixel pairs with a similarity higher than a first preset threshold as target pixel pairs based on the similarity of the dense feature vectors of the two pixels in the pixel pair.

[0016] The acquisition unit is further configured to obtain the initial extrinsic parameters of the camera and the initial three-dimensional point corresponding to each pixel in the target pixel pair based on two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames.

[0017] The projection unit is used to project the initial three-dimensional point corresponding to each pixel in the target pixel pair onto the pixel plane according to the initial extrinsic parameters and the initial intrinsic parameters of the camera, so as to obtain the projection point corresponding to each pixel in the target pixel pair.

[0018] The reconstruction unit is used to construct a loss function based at least on the projection points and the target pixel pair, and to optimize the initial extrinsic parameters of the camera, the initial intrinsic parameters of the camera, and the initial 3D points with the goal of minimizing the loss function, until the camera extrinsic parameters, camera intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

[0019] In a third aspect of this application, an electronic device is provided, comprising: one or more processors;

[0020] Storage device, on which one or more programs are stored,

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional reconstruction method as described in the first aspect.

[0022] In a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional reconstruction method as described in the first aspect.

[0023] In a fifth aspect of this application, a computer program product is provided that, when executed by a processor, implements the three-dimensional reconstruction method as described in the first aspect.

[0024] Therefore, this application has the following beneficial effects:

[0025] The technical solution provided in this application, for each image frame in the video to be processed, first extracts the sparse feature vector and dense feature vector of the pixels of the image frame. For each image frame, based on the comparison between the sparse feature vector of the pixels of the image frame and the sparse feature vector of the pixels of the image frame to be matched (i.e., a certain frame among the image frames to be matched that satisfy a preset adjacent relationship with the current frame), determines the matching pixel pair between the two frames.

[0026] Next, the dense feature vectors of each pixel in these pixel pairs are used to evaluate their similarity, and pixel pairs with similarity higher than a first preset threshold are selected as target pixel pairs. This is done because dense feature vectors can provide richer, higher-dimensional information than sparse features, which helps improve matching accuracy.

[0027] Based on two adjacent image frames in the video to be processed and their corresponding pairs of target pixels, the initial extrinsic parameters of the camera and the preliminary 3D coordinates (initial 3D points) of each pixel in these target pixel pairs are calculated. Then, using the known intrinsic parameters of the camera and the calculated extrinsic parameters, these initial 3D points are projected back onto the pixel plane to form corresponding projection points.

[0028] Next, a loss function is constructed to minimize the difference between the actually observed target pixel and its projected position on the image. This function is then used to iteratively optimize the camera's extrinsic and intrinsic parameters, as well as all relevant 3D points, until the optimal estimate that minimizes the loss function is found. The final results include optimized camera extrinsic and intrinsic parameters, and accurate 3D point cloud data, used to complete high-quality 3D reconstruction.

[0029] This application's technical solution combines the advantages of sparse and dense features: first, sparse features are used to quickly determine potential matching points, and then dense features rich in more information are used to verify and refine the matching results. Even without additional information support from high-precision positioning equipment, this solution can ensure accurate extraction of key point pairs from the image frame sequence of a video, and generate an accurate and reliable 3D model accordingly, reducing the cost of 3D reconstruction while maintaining high accuracy. Attached Figure Description

[0030] Figure 1 A flowchart of a three-dimensional reconstruction method provided in this application embodiment;

[0031] Figure 2 A schematic diagram of a three-dimensional reconstruction technology framework provided in an embodiment of this application;

[0032] Figure 3 A structural diagram of a three-dimensional reconstruction device provided in an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0034] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] To facilitate understanding and explanation of the technical solutions provided in the embodiments of this application, the existing technologies involved in this application will be described first below.

[0036] 3D reconstruction technology is the process of reconstructing a 3D model of an object / structure from a series of 2D images. This process mainly includes the following steps:

[0037] Feature point detection: Using feature detection algorithms such as SIFT, SURF, or ORB, representative feature points are identified in each image. These feature points are typically information-rich regions in the image that are relatively stable across different viewpoints.

[0038] Feature matching: Finding corresponding feature points between pairs of images. This step is done by comparing descriptors between feature points, with the aim of identifying points at the same physical location between two or more images.

[0039] Geometric constraint filtering: To improve matching accuracy, specific geometric constraints (such as epipolar constraints) are applied to filter out matching points that do not meet the conditions. This helps to remove incorrect matches and retain high-quality correspondences.

[0040] Structure from Motion (SfM): Using selected matching points and their geometric relationships, SfM technology estimates the camera parameters (including internal parameters such as focal length and principal point coordinates, as well as external parameters of position and orientation relative to the world coordinate system) for each image frame. The SfM method can recover the 3D structure of a scene and the camera's motion trajectory from a series of 2D images.

[0041] Triangulation: Based on known camera parameters, a preliminary 3D model is constructed by finding corresponding feature points (i.e., feature points that can be identified as the same physical point in images taken from different perspectives) in multiple views and using the specific coordinates in the triangulation.

[0042] Bundle Adjustment (BA): This method uses BA optimization to calculate the positions of these points in a 3D space algorithm, simultaneously adjusting all camera parameters and the positions of the 3D points. The goal is to minimize the reprojection error of all observation data. This process helps to further refine the 3D reconstruction results and improve overall accuracy.

[0043] When using video for 3D reconstruction, in addition to following the basic process described above, the following processing methods also need to be considered:

[0044] A 3D reconstruction method based on image stitching and fusion: This method first analyzes and synthesizes consecutive frames of images from a video, then extracts feature points from the synthesized images, and determines the spatial location of objects by matching these feature points, ultimately constructing a 3D model. However, a major challenge of this method is that distortion may occur during the image stitching process, which affects the quality of feature points and thus reduces the accuracy of the final reconstructed 3D model.

[0045] A 3D reconstruction method combining RTK positioning information: This method combines video with real-time kinematic (RTK) positioning information for 3D reconstruction. This method can provide higher accuracy because it relies on precise geographic location data provided by the RTK device. However, the application scope of this approach is somewhat limited because it requires specific RTK hardware support, and the cost of related equipment is relatively high, making it unsuitable for application scenarios seeking cost-effective solutions.

[0046] Based on this, this application proposes a three-dimensional reconstruction method. For each image frame in the video to be processed, the sparse feature vector and dense feature vector of the pixels in that image frame are extracted. For each image frame, a matching is first performed based on the sparse feature vector of the pixels in that image frame and the sparse feature vector of the pixels in the image frame to be matched, obtaining matching pixel pairs between the two image frames. Then, based on the similarity of the dense feature vectors of the two pixels in these pixel pairs, pixel pairs with a similarity higher than a first preset threshold are selected as target pixel pairs. Since dense feature vectors can provide richer and higher-dimensional information than sparse feature vectors, the matching accuracy is improved.

[0047] Then, using two adjacent image frames of the video to be processed and all target pixel pairs corresponding to those two adjacent image frames, the camera extrinsic parameters and the initial 3D point corresponding to each pixel in these target pixel pairs are calculated. Then, using the known camera intrinsic parameters and the camera extrinsic parameters calculated above, the initial 3D points are projected back onto the pixel plane to obtain the corresponding projection points.

[0048] Finally, by constructing a loss function that indicates the difference between the actual observed target pixel and its projection position on the image, and minimizing this function, the camera extrinsic parameters, camera intrinsic parameters, and all relevant 3D points are iteratively optimized until the best estimate that minimizes the loss function is found, thus completing a high-quality 3D reconstruction.

[0049] As can be seen, the technical solution of this application can ensure the accurate extraction of key point pairs from the image frame sequence of a video, even without the additional information support provided by a high-precision positioning device, and generate an accurate and reliable 3D model accordingly, thereby reducing the cost of 3D reconstruction while maintaining high accuracy.

[0050] It should be noted that the video to be processed in this application can be a video taken in the air when an aircraft (such as a drone) flies around a real-world object (such as various buildings (towers, buildings, archways, etc.), trees, etc.) (i.e., aerial video), or it can be a video taken of the aforementioned object by other devices with video shooting capabilities, which records the complete three-dimensional form of the object.

[0051] To facilitate understanding of the technical solution of this application, the technical terms involved in this application will be explained below.

[0052] Sparse features refer to features where only a few elements in a feature vector are non-zero. These features are commonly found in image processing, text analysis, and recommender systems. Utilizing sparse features can effectively improve computational efficiency and reduce storage requirements because only the non-zero feature values ​​and their positions need to be considered, without operating on the entire vector. This approach is particularly suitable for tasks such as matching and classification on large-scale datasets. However, sparse representation also has potential drawbacks: it may lead to the loss of some detailed information from the original data, thus affecting the model's ability to recognize subtle differences or its overall performance.

[0053] Dense features refer to feature representations where most elements in the feature vector have non-zero values. This type of feature is commonly used in tasks such as image matching and image segmentation. Due to their high information content, dense features can capture richer details, such as color, texture, and shape, making tasks based on these features (e.g., pixel-level matching) more accurate and reliable. However, precisely because of their high data density, processing these features is computationally expensive. In fine-grained image matching, utilizing the detailed description of each pixel provided by dense features can help reduce false matches caused by insufficient features.

[0054] Camera intrinsic parameters refer to a set of parameters describing the internal optical and geometric characteristics of a camera. These mainly include focal length, the position coordinates of the principal point (i.e., the point in the pixel plane where the optical axis intersects with the principal point), and distortion coefficients. These parameters reflect the relationship between the camera lens and the imaging sensor and are determined by the camera's design and manufacturing process. Theoretically, they will not change over time without external physical damage.

[0055] The camera extrinsic parameters, also known as the extrinsic parameter matrix (world to camera, w2c), describe the camera's position and orientation relative to the world coordinate system. It consists of a 3×3 rotation matrix R and a 3×1 translation vector t, which together define the transformation from the world coordinate system to the camera coordinate system. Specifically, the rotation matrix R represents the camera's rotation angle around the world coordinate axes; while the translation vector t indicates the camera's position relative to the world origin. Therefore, the extrinsic parameter matrix can be used to transform 3D points in the world coordinate system to the camera coordinate system. Camera extrinsic parameter matrix P w2c It can be written as:

[0056] Pw2c = [R | t] where | represents the concatenation operation, meaning that the matrix is ​​actually a 4×4 form that contains rotation and translation information, even though the core part is 3×4.

[0057] The inverse of the camera extrinsic parameter matrix is ​​called the (camera-to-world, c2w) matrix, or camera pose. This matrix has the opposite function: transforming points in the camera coordinate system back to the world coordinate system. Its form is:

[0058]

[0059] Among them, R T It is the transpose of R, because R is an orthonormal identity matrix, and its inverse is equal to its transpose.

[0060] The process of transforming from the world coordinate system to the camera coordinate system involves using camera extrinsic parameters; while the further transformation from the camera coordinate system to the pixel plane (pixel coordinate system) requires the use of camera intrinsic parameters. Camera intrinsic parameters are related to the camera's inherent characteristics, such as focal length and principal point offset. These parameters define how points in three-dimensional space are projected onto a two-dimensional image. The pixel coordinate system refers to the coordinate system used to represent the position of each pixel in a digital image.

[0061] Co-viewing relationship refers to a situation in computer vision or SLAM applications where two or more image keyframes jointly observe the same object or scene portion, thus forming a co-viewing relationship between these image keyframes.

[0062] Local features of an image refer to unique, stable, and repeatable local structures or patterns that allow for reliable detection even under varying scales, rotation angles, and lighting conditions. A typical local feature usually consists of two parts: first, keypoints in the image (such as corners), which are locations that provide sufficient information to distinguish different regions; and second, a feature descriptor extracted from a small region associated with the keypoint, describing the specific attributes of that region. Local features are widely used in computer vision tasks, including but not limited to object recognition, image matching, classification, and tracking. Common types of local features include corners (such as Harris corners), edges (detected using methods such as the Canny operator), and textures (represented using techniques such as LBP-Local Binary Patterns).

[0063] A corner is a location in an image that has a sharp angle or bend, usually formed by the intersection of two edges at a certain angle. The pixel values ​​around a corner show significant variations in different directions, indicating that the region contains rich local structural information. Compared to edge features, corner features have more advantages in terms of discriminative power and information content. Generally, a corner is not a single pixel, but rather determined by a group of pixels in its surrounding neighborhood, which exhibit strong directional changes. Therefore, a corner actually refers to a set of pixels within a specific region. However, in some special cases, if an image contains extremely sharp corners or bends, a single pixel may be sufficient to represent the entire corner.

[0064] The technical solution provided in this application will be described below with reference to the accompanying drawings.

[0065] See Figure 1 The figure is a flowchart of a three-dimensional reconstruction method provided in the first embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0066] S101: For image frames in the video to be processed, extract the sparse feature vector and dense feature vector of the pixels in the image frame.

[0067] After acquiring the video to be processed, according to one embodiment of this application, feature vectors (including sparse and dense feature vectors) of pixels can be extracted for each frame of the video (i.e., image frame). In another embodiment, to improve the efficiency of the 3D reconstruction process while maintaining reconstruction quality, feature vectors can be extracted only from selected keyframes. Keyframes refer to specific image frames that provide important information about the position, rotation angle, etc., of objects in the real world. Whether processing each frame or a keyframe, one of two methods can be chosen when extracting feature vectors: one is to perform this operation for every pixel in the image frame; the other is to select only certain feature pixels (referred to as feature points) within the image frame. The feature points mentioned here usually refer to pixels or corners that are stable under different viewing angles, such as edges, corners, and other points that are easy to identify and do not easily change with the viewing angle.

[0068] In practical applications, a trained deep learning model can be used to extract sparse and dense feature vectors of pixels in an image frame, respectively. The specific steps are as follows:

[0069] Sparse Feature Vector Extraction: This method utilizes a pre-trained neural network (such as a convolutional neural network) specifically designed for extracting sparse feature vectors, taking an image frame as input. The network can identify and output sparse feature vectors corresponding to key points or salient regions within the image frame.

[0070] Dense Feature Vector Extraction: Each image frame is fed into another pre-trained neural network that generates dense feature representations (a convolutional neural network or other suitable architecture can also be chosen). The result of this step is a feature map containing feature descriptors for each pixel location, i.e., a dense feature vector. After obtaining such a feature map, for any specific pixel in the original image frame, the corresponding dense feature vector can be directly located from the feature map based on its coordinates.

[0071] In this application, there are no strict restrictions on the type of neural network used to extract sparse or dense features; they can all be built based on convolutional operations, or other types of network architectures can be selected according to actual needs.

[0072] S102: For each image frame, based on the sparse feature vector of the pixels in the image frame and the sparse feature vector of the pixels in the image frame to be matched, determine the pixel pair formed by the matching pixels of the image frame and the image frame to be matched.

[0073] It is understandable that a pixel pair consists of two pixels, one from the image frame and the other from the image frame to be matched.

[0074] The image frame to be matched refers to an image frame in the video to be processed that satisfies a preset adjacency relationship with the image frame. Specifically, the adjacency relationship can be N consecutive image frames in the video to be processed that follow the image frame, where N is an integer greater than or equal to 1.

[0075] For example, the image frame to be matched refers to the 20 consecutive image frames following the image frame in the video to be processed. Then, determining the pixel pairs formed by the matching pixels of the image frame and the image frame to be matched, based on the sparse feature vectors of the pixels in the image frame and the image frame to be matched, can be understood as follows:

[0076] Based on the sparse feature vector of the pixels in the image frame and the sparse feature vector of the pixels in each of the 20 image frames to be matched, the pixel pairs formed by the matching pixels of the image frame and each of the 20 image frames to be matched are determined.

[0077] Assuming that the image frame has 20 matching pixel pairs with the first image frame to be matched, 10 matching pixel pairs with the third image frame to be matched, 7 matching pixel pairs with the fifth image frame to be matched, and 3 matching pixel pairs with the sixth image frame to be matched, then the total number of pixel pairs formed by the matching pixels of the image frame and the image frames to be matched, as determined in step S102, is 40.

[0078] It should be noted that the number of image frames to be matched may vary for different image frames in the video to be processed. That is, N is a variable value, and its specific value depends on the total number of image frames in the video and its initial value. For example, if the video to be processed contains 21 image frames, and the initial value of N is 20, then for the first image frame, the corresponding image frames to be matched are the 20 image frames following it; for the second image frame, the corresponding image frames to be matched are the 19 image frames following it; for the third image frame, the corresponding image frames to be matched are the 18 image frames following it, and so on, with the second-to-last image frame being the last image frame to be matched.

[0079] S103: Based on the similarity of the dense feature vectors of two pixels in a pixel pair, select pixel pairs with a similarity higher than a first preset threshold as target pixel pairs.

[0080] Considering the limited image information provided by sparse feature vectors, mismatches may exist in pixel pairs determined based on sparse feature vector similarity. To avoid these mismatches affecting the quality of 3D reconstruction, this application eliminates erroneous matches through step S103. Specifically, for each pixel pair, the similarity of its corresponding dense feature vector is calculated. If the similarity of the dense feature vector of a pixel pair is lower than a first preset threshold, the pixel pair is considered an erroneous match and is eliminated (i.e., the matching relationship between the pixel pair is terminated). Pixel pairs with a dense feature vector similarity higher than the first preset threshold are retained as target matching points. The similarity between dense feature vectors can be calculated using cosine similarity or Euclidean distance, etc.

[0081] In other words, this embodiment utilizes the dense feature vectors of the pixels when determining the matching pixels between two image frames. Since dense feature vectors can provide richer and higher-dimensional feature descriptions, this helps to improve the accuracy of matching, thereby ensuring that high-quality camera intrinsic and extrinsic parameter estimation and 3D reconstruction results can be obtained based on these matching points.

[0082] S104: Based on the two adjacent image frames of the video to be processed and all target pixel pairs corresponding to the two adjacent image frames, obtain the initial extrinsic parameters of the camera and the initial 3D point corresponding to each pixel in the target pixel pair.

[0083] S105: Based on the initial extrinsic and intrinsic parameters of the camera, project the initial 3D point corresponding to each pixel in the target pixel pair onto the pixel plane to obtain the projection point corresponding to each pixel in the target pixel pair.

[0084] S106: Construct a loss function based at least on the projection point and the target pixel pair. With the goal of minimizing the loss function, optimize the initial extrinsic parameters, initial intrinsic parameters, and initial 3D points of the camera until the camera extrinsic parameters, intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

[0085] Wherein, constructing a loss function based at least on the projection point and the target pixel pair means that, in one embodiment, the input to constructing the loss function may only include the projection point and the target pixel pair.

[0086] Ideally, given the 3D coordinates of a pixel, when projected onto the pixel plane using the correct camera extrinsic and intrinsic parameters, its projected position should perfectly coincide with the actually observed pixel. However, in the real world, due to various factors such as measurement errors and environmental interference, there is often a deviation between the projected point and the actual pixel. Therefore, by comparing the positions of all known pixels and their corresponding 3D points projected using the currently estimated camera parameters (including extrinsic and intrinsic parameters), a loss function can be defined to quantify the degree of this deviation. Then, by iteratively adjusting the camera's extrinsic and intrinsic parameters, as well as the point coordinates in 3D space, the aim is to reduce or even minimize this loss function value, thereby achieving accurate 3D scene reconstruction.

[0087] The above is a 3D reconstruction scheme provided by an embodiment of this application. This scheme first uses sparse features to quickly determine potential matching points, and then uses dense features rich in more information to verify and improve the matching results. Through this scheme, even without additional information support provided by high-precision positioning equipment, it can ensure accurate extraction of key point pairs from the image frame sequence of a video, and generate an accurate and reliable 3D model accordingly, thereby reducing the cost of 3D reconstruction while maintaining high accuracy.

[0088] Considering that dense feature vectors can provide more information, in order to improve the quality of 3D reconstruction, in the second embodiment provided in this application, in Figure 1 Before constructing the loss function as shown, the following steps can be further performed:

[0089] Obtain the dense feature vector of the projection point corresponding to each pixel in the target pixel pair;

[0090] Obtain the similarity between the dense feature vectors of the projected point and the corresponding target pixel point pair.

[0091] For example, if pixel 1 and pixel 2 constitute a target pixel pair, pixel 1 corresponds to projection point n, and pixel 2 corresponds to projection point m, then calculate the dense feature vector similarity between pixel 1 and projection point n, and calculate the dense feature vector similarity between pixel 2 and projection point m.

[0092] In this embodiment, the construction of the loss function in S106, based at least on the projection points and the target pixel points, can be implemented as follows:

[0093] A loss function is constructed based on the similarity between the projection point, the target pixel pair, and the dense feature vectors of the pixels in the corresponding target pixel pair.

[0094] Since errors exist in the feature matching process, the accumulated error becomes larger and larger as the number of feature matches increases. To address this, this application proposes to project 3D points onto the pixel plane according to the camera pose, and then calculate the similarity of dense features between the projected points and their corresponding pixels as a loss function. By optimizing this loss function, global accumulated error can be eliminated, further improving the effect of 3D reconstruction.

[0095] Considering that aerial videos taken by aircraft are often far from the surface of objects in the scene, in order to further improve the quality of 3D reconstruction, this application provides a third embodiment based on the first and second embodiments described above. The difference between this embodiment and the previous embodiments is that, between S103 and S104, the following steps are also included:

[0096] For each image frame, based on the dense feature vector of each pixel in the target pixel pair corresponding to the image frame and the image frame to be matched, find the pixel with the highest similarity to the dense feature vector of the pixel corresponding to the target pixel in the image frame and the image frame to be matched, and update the pixel in the target pixel pair to the pixel with the highest similarity to the dense feature vector.

[0097] This method ensures that the two pixels in a target pixel pair determined from aerial video are the two pixels with the highest similarity in their dense feature vectors. For example, if the target pixel pair includes a first pixel and a second pixel, where the first pixel comes from a certain image frame and the second pixel comes from a certain image frame to be matched, based on the dense feature vector of the first pixel, a third pixel with the highest similarity in the image frame to be matched is found, and the second pixel in the target pixel pair is updated to this third pixel. It is understood that the third pixel may be the second pixel, or it may be a pixel located adjacent to the second pixel.

[0098] Although the pixel pairs determined by S102 are prone to inaccurate matching, considering that S102 is based on the similarity of the sparse feature vectors of two pixels, the pixel that truly matches the first pixel is usually not too far from the second pixel. Therefore, to quickly find the third pixel that matches the first pixel, a search is performed within a preset pixel space corresponding to the second pixel. The pixel with the highest similarity to the dense feature vector of the first pixel within this preset pixel space is determined as the third pixel. For example, with the second pixel as the center, the search is performed within a certain radius to find the third pixel with the highest similarity to the dense feature vector of the first pixel.

[0099] It should be noted that since a single image frame can have multiple image frames to be matched, the above update operation is performed for each image frame to be matched, so as to maximize the similarity of the dense feature vectors between the target pixel pairs formed by the image frame and each image frame to be matched.

[0100] The above are three embodiments of three-dimensional reconstruction provided in this application. The following is a detailed description of the specific implementation methods of some technical means in the above three embodiments.

[0101] First, in S104, the step of obtaining the initial extrinsic parameters of the camera and the initial 3D point corresponding to each pixel in the target pixel pair based on two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames can be implemented as follows:

[0102] Select the two image frames with the largest parallax from all adjacent image frames of the video to be processed;

[0103] Based on the two image frames with the largest parallax and all target pixel pairs corresponding to the two image frames, the initial external parameters of the camera corresponding to the two image frames with the largest parallax are obtained.

[0104] Using the camera's initial extrinsic and initial intrinsic parameters, triangulation is performed on each pixel in the target pixel pair corresponding to the two image frames with the largest parallax to obtain the initial three-dimensional point corresponding to each pixel in the target pixel pair.

[0105] For the image frames in the video to be processed, except for the two image frames with the largest parallax, the other image frames are aligned with the two image frames with the largest parallax. The (Perspective-n-Point, PNP) algorithm is used to obtain the initial extrinsic parameters of the camera corresponding to the other image frames and the initial 3D point corresponding to each pixel in the target pixel pair of the other image frames.

[0106] Parallax refers to the difference in the position of an object in an image when it is photographed from different viewpoints. The two frames with the largest parallax indicate the greatest difference in the object's position, providing more information for calculating the camera's initial extrinsic parameters and improving the stability of the determined initial parameters. The Perspective-n-Point (PNP) algorithm is primarily used to solve the pose problem of the camera coordinate system relative to the world coordinate system. When the world coordinates of multiple 3D points and their corresponding pixel coordinates in the image are known, the PNP algorithm can estimate the camera pose, i.e., solve for the rotation matrix and translation vector from the world coordinate system to the camera coordinate system.

[0107] In practical implementation, the initial extrinsic parameters of the camera are obtained based on the two image frames with the largest parallax and all target pixel pairs corresponding to those two image frames. This can be achieved in the following way:

[0108] Based on the two image frames with the largest disparity and all target pixel pairs corresponding to these two image frames, the transformation matrix of these two image frames is obtained. Specifically, based on the two image frames with the largest disparity and all target pixel pairs corresponding to them, the transformation matrix from one viewpoint to another is calculated using RANSAC+ fundamental matrix / essential matrix estimation, direct linear transformation (DLT), or the 8-point algorithm. This transformation matrix reflects the relative pose change between the camera positions when these two image frames were captured.

[0109] The transformation matrices of the two image frames are decomposed to obtain the corresponding camera extrinsic parameters, which are then used as the initial camera extrinsic parameters. Specifically, the initial camera extrinsic parameters (i.e., rotation matrix R and translation vector t) between the two image frames can be obtained through singular value decomposition (SVD) or other methods.

[0110] In another embodiment of this application, the transformation matrix corresponding to the two adjacent image frames can also be determined based on all target pixel pairs corresponding to the two adjacent image frames that meet other conditions, and the transformation matrix can be decomposed to obtain the camera extrinsic parameters corresponding to the two image frames as the initial extrinsic parameters of the camera. This application does not limit this.

[0111] To further improve the quality of 3D modeling, in another embodiment of this application, after obtaining the transformation matrix and before projecting the initial 3D point corresponding to each pixel in the target pixel pair onto the pixel plane, the following steps may be further included:

[0112] Based on the transformation matrix corresponding to the two adjacent image frames with the largest disparity, a consistency check is performed on the two pixels in the target pixel pair, and the target pixel pair that fails the consistency check is deleted.

[0113] Specifically, assume we have two adjacent image frames with the largest disparity, image frame 1 and image frame 2, and a transformation matrix H between these two frames. Now consider a pair of matching target pixels: pixel P1 in image frame 1 and pixel P2 in image frame 2. First, use the transformation matrix H to transform pixel P1 in image frame 1 to the spatial coordinate system of image frame 2, calculating the new position P1' corresponding to P1. Check whether the new position P1' is consistent with or sufficiently close to pixel P2 in image frame 2. "Sufficiently close" usually means that the distance between P1' and P2 is less than a preset threshold. If this condition is met, the pair of pixels (P1, P2) is considered consistent, meaning they correctly represent points in the same physical world; otherwise, the pair of pixels is considered mismatched and should be excluded from subsequent processing. The above steps need to be repeated for all target pixel pairs until the consistency verification of all pixels is completed. This method can effectively remove erroneous matches caused by noise, occlusion, or other factors, thereby improving the quality of the final 3D reconstruction model.

[0114] Secondly, for image frame S102 and an image frame to be matched, based on the sparse feature vectors of the pixels in the image frame and the sparse feature vectors of the pixels in the image frame to be matched, the pixel pairs formed by the matching pixels of the image frame and the image frame to be matched are determined. This can be specifically implemented as follows:

[0115] Calculate the sparse feature vector similarity between the sparse feature vector of the pixels in the image frame and the sparse feature vector of the pixels in the image frame to be matched; wherein, the pixels can be all the pixels in the image frame or selected pixels, see the relevant section above for details; the method for calculating the sparse feature vector similarity can be cosine similarity, Euclidean distance, etc.

[0116] Two pixels whose sparse feature vector similarity is greater than a second preset threshold are identified as matching pixel pairs. If the sparse feature vector similarity between a pair of pixels exceeds the second preset threshold, the pair of pixels is considered to be matched and they are marked as a single pixel pair.

[0117] For example, the sparse feature vector of pixel 1 in the image frame is X1, and the sparse feature vector of pixel 2 in a certain image frame to be matched is X2. The similarity P between the sparse feature vectors of X1 and X2 is calculated by the cosine similarity method. If the similarity P is greater than the second preset threshold, then pixel 1 and pixel 2 are determined to be matched pixels, and pixel 1 and pixel 2 constitute a pixel pair.

[0118] Typically, sparse feature matching involves sequentially matching image frames with multiple image frames to be matched. To improve matching accuracy, neural networks can be used to perform sparse feature matching between two image frames. Specifically, the sparse feature vectors of pixels in one image frame and the corresponding sparse feature vectors of pixels in another image frame (to be matched) are input into a pre-trained neural network. This network outputs pixel pairs whose sparse feature vector similarity exceeds a second preset threshold between the two image frames.

[0119] In practical implementation, the neural network used to perform the sparse feature matching task and the neural network used to extract sparse feature vectors can be the same model. Through proper training, this neural network can not only effectively extract sparse feature vectors from image frames, but also identify matching pixel pairs based on the sparse feature vectors between different image frames.

[0120] To facilitate understanding of the technical implementation of this application, the technical principles of the preferred embodiment of the three-dimensional reconstruction technology of this application are introduced below in conjunction with aerial photography videos, such as... Figure 2 The diagram shown illustrates the framework of the three-dimensional reconstruction technology. The three-dimensional reconstruction process provided in this application includes:

[0121] (1) Feature vector extraction, including:

[0122] (1.1) Sparse Feature Vector Extraction: For image frames in the video to be processed, extract the positions of pixels that do not change in spatial scale, and encode a description of their invariance to form a descriptor. Specifically, SuperPoint can be used to extract sparse feature vectors. Since aerial data scenes are relatively complex, features extracted by using convolutional neural networks are more robust than traditional manually designed features.

[0123] (1.2) Dense Feature Vector Extraction; Since the pixel intensity of an image is sensitive to changes in factors such as viewing angle and illumination, this application can use deep learning methods to extract dense feature vectors. For each frame of the image, a D-dimensional feature map is obtained through a convolutional neural network. The length and width of this feature map are the same as the original image. Therefore, a D-dimensional dense feature vector can be obtained for each pixel position based on the feature map.

[0124] (2) Feature matching and feature improvement

[0125] (2.1) Sparse feature matching

[0126] Specifically, feature matching is performed between different image frames based on the sparse feature vectors extracted in section 1.1. Since the image frames in the aerial video have a sequential relationship, this application adopts a sequential matching method, matching each image frame with its N neighboring image frames. When matching two image frames, SuperGlue can be used for feature matching in practice. Because neural networks can extract information from a wider range of images, matching based on feature point descriptors using neural networks achieves higher accuracy.

[0127] (2.2) False Match Removal

[0128] Specifically, for the feature matching results obtained in 2.1, the dense feature vectors corresponding to the two pixels in the matching pair are compared. If the similarity of the dense feature vectors is low, the task matching is incorrect and is removed. Then, geometric verification is performed on the two matching images. RANSAC is used to find the interior points that conform to the essential matrix or homography matrix, and other incorrect matches of exterior points are removed.

[0129] (2.3) Feature location improvement (re-matching)

[0130] Specifically, since aerial videos are often far from the surface of objects in the scene, a misalignment of even one pixel can lead to a significant positional shift in the reconstruction result. Therefore, for the feature matching results after removing incorrect matches in 2.2, this application searches within a certain pixel space around the dense feature vector to find the position with the highest similarity between the dense feature vectors of the feature pixels, and then moves the sparse feature positions extracted in 2.1.

[0131] (3) Image registration and global BA

[0132] (3.1) Image Registration

[0133] Specifically, based on the feature matching results obtained in step 2, the transformation matrix between two image frames is first calculated at the location with large disparity, and the features are triangulated to obtain the initial three-dimensional points in the initial three-dimensional model. Then, according to the co-view relationship between image frames, new image frame observations are added to the three-dimensional model in sequence, and new initial three-dimensional points are generated by triangulation.

[0134] (3.2) Global BA

[0135] Because errors exist during image registration, and these errors accumulate significantly with the increasing number of registered images, global basis analysis (BA) is used to optimize camera pose and 3D points. This application proposes a novel BA method that projects 3D points onto a pixel plane based on the camera pose, then calculates the similarity between the dense feature vectors of the projected points and their corresponding 2D feature points as a loss function. Optimizing this loss function eliminates the global accumulated error.

[0136] Based on the above method embodiments, this application provides a three-dimensional reconstruction device and an electronic device, which will be described below with reference to the accompanying drawings.

[0137] See Figure 3 This figure is a structural diagram of a three-dimensional reconstruction device provided in an embodiment of this application, such as... Figure 3 As shown, the device 300 includes: an acquisition unit 301, a projection unit 302, and a reconstruction unit 303.

[0138] The acquisition unit 301 is used to extract the sparse feature vector and dense feature vector of the pixels of the image frame for the video to be processed.

[0139] The acquisition unit 301 is further configured to, for each image frame, determine the pixel pair formed by the matching pixels of the image frame and the image frame to be matched, based on the sparse feature vector of the pixels of the image frame and the sparse feature vector of the pixels of the image frame to be matched, wherein the image frame to be matched is an image frame in the video to be processed that satisfies a preset adjacent relationship with the image frame.

[0140] The acquisition unit 301 is further configured to select pixel pairs with a similarity higher than a first preset threshold as target pixel pairs based on the similarity of the dense feature vectors of the two pixels in the pixel pair.

[0141] The acquisition unit 301 is also used to obtain the initial external parameters of the camera and the initial three-dimensional point corresponding to each pixel in the target pixel pair based on the two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames.

[0142] The projection unit 302 is used to project the initial three-dimensional point corresponding to each pixel in the target pixel pair onto the pixel plane according to the initial extrinsic parameters and the initial intrinsic parameters of the camera, so as to obtain the projection point corresponding to each pixel in the target pixel pair.

[0143] The reconstruction unit 303 is used to construct a loss function based at least on the projection points and the target pixel points, and to optimize the initial extrinsic parameters of the camera, the initial intrinsic parameters of the camera, and the initial 3D points with the goal of minimizing the loss function, until the camera extrinsic parameters, camera intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

[0144] In some embodiments, the acquisition unit 301 is further configured to acquire the dense feature vector of the projection point; and acquire the similarity between the dense feature vector of the projection point and each pixel in its corresponding target pixel pair;

[0145] The reconstruction unit 303 is specifically used to construct a loss function based on the similarity between the projection point, the target pixel pair, and the dense feature vector of each pixel in the corresponding target pixel pair.

[0146] In some embodiments, the apparatus further includes: an updating unit;

[0147] The updating unit is configured to, before the acquisition unit 301 obtains the initial extrinsic parameters of the camera and the initial 3D point corresponding to each pixel in the target pixel pair based on the two adjacent image frames of the video to be processed and all target pixel pairs corresponding to the two adjacent image frames, for each image frame, based on the dense feature vector of the pixels in the target pixel pair corresponding to the image frame and the image frame to be matched, find the pixel with the highest similarity to the dense feature vector of the pixels in the target pixel pair in the image frame and the image frame to be matched, and update the pixels in the target pixel pair to the pixel with the highest similarity to the dense feature vector.

[0148] In some embodiments, the acquisition unit 301 is specifically used to select the two image frames with the largest disparity from all adjacent image frames; based on the two image frames with the largest disparity and all target pixel pairs corresponding to the two image frames, obtain the initial extrinsic parameters of the camera corresponding to the two image frames with the largest disparity; using the initial extrinsic parameters and the initial intrinsic parameters of the camera, perform triangulation processing on each pixel in the target pixel pairs corresponding to the two image frames with the largest disparity to obtain the initial three-dimensional point corresponding to each pixel in the target pixel pairs; for other image frames in the video to be processed, align the other image frames to the two image frames with the largest disparity, and obtain the initial extrinsic parameters of the camera corresponding to the other image frames and the initial three-dimensional point corresponding to each pixel in the target pixel pairs corresponding to the other image frames through the PNP algorithm.

[0149] In some embodiments, the acquisition unit 301 is specifically used to obtain the transformation matrix of the two image frames with the largest disparity and all target pixel pairs corresponding to the two image frames; and to decompose the transformation matrix of the two image frames to obtain the camera extrinsic parameters corresponding to the two image frames as the initial extrinsic parameters of the camera.

[0150] In some embodiments, the apparatus further includes: a deletion unit;

[0151] The deletion unit is used to perform a consistency check on two pixels in the target pixel pair before the projection unit 302 projects the initial three-dimensional point corresponding to each pixel in the target pixel pair onto the pixel plane, based on the transformation matrix corresponding to the two adjacent image frames with the largest disparity, and delete the target pixel pair that fails the consistency check.

[0152] In some embodiments, the acquisition unit 301 is specifically used to acquire a feature map of each image frame of the video to be processed using a trained neural network. The feature map records the dense feature vectors corresponding to the pixels of the image frame.

[0153] It should be noted that the specific implementation of each unit in this embodiment can be found in the relevant descriptions in the above method embodiments. The division of units in this application embodiment is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. The functional units in this application embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiments, the processing unit and the sending unit can be the same unit or different units. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] Based on the three-dimensional reconstruction method provided in the above-described method embodiments, this application also provides an electronic device, including: one or more processors; and a storage device storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional reconstruction method described in any of the above embodiments.

[0155] The following is for reference. Figure 4 This document illustrates a structural schematic diagram of an electronic device 400 suitable for implementing embodiments of this application. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Android Devices), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 4As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. The processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0157] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0158] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this application.

[0159] The electronic device provided in this application embodiment and the three-dimensional reconstruction method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0160] Based on the three-dimensional reconstruction method provided in the above-described embodiments, this application provides a computer-readable medium storing a computer program thereon, wherein the program, when executed by a processor, implements the three-dimensional reconstruction method as described in any of the above embodiments.

[0161] It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0162] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0163] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0164] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned three-dimensional reconstruction method.

[0165] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] The units described in the embodiments of this application can be implemented in software or in hardware. The name of the unit / module does not necessarily limit the unit itself; for example, a voice data acquisition module can also be described as a "data acquisition module".

[0168] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0170] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0171] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0172] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0173] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: For the image frames of the video to be processed, extract the sparse feature vectors and dense feature vectors of the pixels in the image frames; For each image frame, based on the sparse feature vector of the pixels in the image frame and the sparse feature vector of the pixels in the image frame to be matched, a pixel pair consisting of the pixels of the image frame and the image frame to be matched is determined. The image frame to be matched is an image frame in the video to be processed that satisfies a preset adjacent relationship with the image frame. Based on the similarity of the dense feature vectors of the two pixels in the pixel pair, pixel pairs with a similarity higher than a first preset threshold are selected as target pixel pairs. Based on the two adjacent image frames of the video to be processed and all target pixel pairs corresponding to the two adjacent image frames, the initial extrinsic parameters of the camera and the initial three-dimensional point corresponding to each pixel in the target pixel pair are obtained. Based on the initial extrinsic and initial intrinsic parameters of the camera, the initial 3D point corresponding to each pixel in the target pixel pair is projected onto the pixel plane to obtain the projection point corresponding to each pixel in the target pixel pair. A loss function is constructed based at least on the projection points and the target pixel pair. With the goal of minimizing the loss function, the initial extrinsic parameters of the camera, the initial intrinsic parameters of the camera, and the initial 3D points are optimized until the camera extrinsic parameters, camera intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the dense feature vector of the projection points; Obtain the similarity between the projection point and the dense feature vector of each pixel in its corresponding target pixel pair; The construction of the loss function based at least on the projection point and the target pixel pair specifically includes: A loss function is constructed based on the similarity between the projection point, the target pixel pair, and the dense feature vector of each pixel in the corresponding target pixel pair.

3. The method according to claim 1, characterized in that, Before obtaining the initial extrinsic parameters of the camera and the initial 3D point corresponding to each pixel in the target pixel pair based on two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames, the method further includes: For each image frame, based on the dense feature vector of the pixels in the target pixel pair corresponding to the image frame and the image frame to be matched, find the pixel with the highest similarity to the dense feature vector of the pixels in the target pixel pair in the image frame and the image frame to be matched, and update the pixels in the target pixel pair to the pixel with the highest similarity to the dense feature vector.

4. The method according to any one of claims 1-3, characterized in that, The process of obtaining the initial extrinsic parameters of the camera and the initial 3D point corresponding to each pixel in the target pixel pair based on two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames specifically includes: Select the two image frames with the largest parallax from all adjacent image frames; Based on the two image frames with the largest parallax and all target pixel pairs corresponding to the two image frames, the initial extrinsic parameters of the camera corresponding to the two image frames with the largest parallax are obtained. Using the initial extrinsic and intrinsic parameters of the camera, triangulation is performed on each pixel in the target pixel pair corresponding to the two image frames with the largest parallax to obtain the initial three-dimensional point corresponding to each pixel in the target pixel pair. For the other image frames in the video to be processed, excluding the two image frames with the largest parallax, the other image frames are aligned with the two image frames with the largest parallax. The PNP algorithm is used to obtain the initial extrinsic parameters of the camera corresponding to the other image frames and the initial 3D point corresponding to each pixel in the target pixel pair corresponding to the other image frames.

5. The method according to claim 4, characterized in that, Based on the two image frames with the largest parallax and all target pixel pairs corresponding to these two image frames, the initial extrinsic parameters of the camera are obtained, including: Based on the two image frames with the largest disparity and all target pixel pairs corresponding to the two image frames, the transformation matrix of the two image frames is obtained. The transformation matrices of the two image frames are decomposed to obtain the camera extrinsic parameters corresponding to the two image frames, which are then used as the initial extrinsic parameters of the camera.

6. The method according to claim 5, characterized in that, Before projecting the initial 3D point corresponding to each pixel in the target pixel pair onto the pixel plane, the method further includes: Based on the transformation matrix corresponding to the two adjacent image frames with the largest disparity, a consistency check is performed on the two pixels in the target pixel pair, and the target pixel pair that fails the consistency check is deleted.

7. The method according to any one of claims 1-3, characterized in that, The step of extracting dense feature vectors of pixels from image frames of the video to be processed includes: For each image frame of the video to be processed, a trained neural network is used to obtain the feature map of that image frame. The feature map records the dense feature vectors corresponding to the pixels of that image frame.

8. A three-dimensional reconstruction device, characterized in that, The device includes: The acquisition unit is used to extract the sparse feature vector and dense feature vector of the pixels of the image frame for the video to be processed; The acquisition unit is further configured to, for each image frame, determine the pixel pair formed by the matching pixels of the image frame and the image frame to be matched, based on the sparse feature vector of the pixels of the image frame and the sparse feature vector of the pixels of the image frame to be matched, wherein the image frame to be matched is an image frame in the video to be processed that satisfies a preset adjacent relationship with the image frame. The acquisition unit is further configured to select pixel pairs with a similarity higher than a first preset threshold as target pixel pairs based on the similarity of the dense feature vectors of the two pixels in the pixel pair. The acquisition unit is further configured to obtain the initial extrinsic parameters of the camera and the initial three-dimensional point corresponding to each pixel in the target pixel pair based on two adjacent image frames of the video to be processed and all target pixel point pairs corresponding to the two adjacent image frames. The projection unit is used to project the initial three-dimensional point corresponding to each pixel in the target pixel pair onto the pixel plane according to the initial extrinsic parameters and the initial intrinsic parameters of the camera, so as to obtain the projection point corresponding to each pixel in the target pixel pair. The reconstruction unit is used to construct a loss function based at least on the projection points and the target pixel pair, and to optimize the initial extrinsic parameters of the camera, the initial intrinsic parameters of the camera, and the initial 3D points with the goal of minimizing the loss function, until the camera extrinsic parameters, camera intrinsic parameters, and 3D points corresponding to the minimum loss function are obtained as the 3D reconstruction result.

9. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the three-dimensional reconstruction method as described in any one of claims 1-7.

Images