Video-based three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a video-based three-dimensional reconstruction method and device, electronic equipment and a storage medium, which are used for performing video three-dimensional reconstruction more accurately and quickly. The method comprises the following steps: for a to-be-processed video, detecting matching feature points and position information of the matching feature points among frames frame by frame, and meanwhile, executing initial reconstruction operation by adopting the following modes: selecting a first key frame and a second key frame from the video according to the number of the matching feature points and inter-frame displacement, based on the poses corresponding to the initial video frames in the range from the first key frame to the second key frame, performing three-dimensional reconstruction on the initial video frames in combination with the position information; and iteratively executing incremental reconstruction operation by adopting the following mode: selecting a third key frame after the second key frame according to the number and inter-frame displacement of each matching feature point, and performing three-dimensional reconstruction on each incremental video frame based on the pose corresponding to each incremental video frame from the second key frame to the third key frame in combination with the position information.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video-based 3D reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of internet technology, augmented reality (AR) technology based on network technology is being widely applied in an increasing number of fields, such as short videos, games, user interface (UI) interaction, mechanical assembly, equipment maintenance, structural display, electronic maps, and navigation. Among these, AR virtual object placement is being used as a new interaction method in multiple applications. Furthermore, the rapid and efficient spatial iterative optimization and reconstruction of videos to support 3D effects such as spatial text is a crucial capability of AR. Therefore, how to perform more accurate and faster 3D reconstruction of videos is currently a hot research topic.

[0003] However, most current 3D reconstruction solutions for videos are based on personal computers (PCs) and the cloud. Because 3D reconstruction requires high computing power and the algorithms are time-consuming, the reconstruction speed is slow. The average time to reconstruct a video of tens of seconds is more than 10 minutes, and the failure rate is also high, making it difficult to implement on mobile devices. Summary of the Invention

[0004] This application provides a video-based 3D reconstruction method, apparatus, electronic device, and storage medium, which proposes a video 3D reconstruction technology that can be implemented on a mobile device, thereby enabling more accurate and faster 3D reconstruction of videos.

[0005] This application provides a video-based 3D reconstruction method, comprising:

[0006] For the video to be processed, each matching feature point between frames and the position information of each matching feature point on the corresponding video frame are detected frame by frame; during the frame-by-frame detection process, the following reconstruction operation is performed based on the detected video frames, and the undetected video frames are further processed:

[0007] The initial reconstruction operation is performed in the following manner: based on the number of each matching feature point and the inter-frame displacement, a first keyframe and a second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame.

[0008] The incremental reconstruction operation is performed iteratively in the following manner until the video processing is completed: based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed on each incremental video frame; wherein, in subsequent iterations other than the first round of incremental reconstruction operation, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

[0009] This application provides a video-based 3D reconstruction device, comprising:

[0010] The feature processing unit is used to detect, frame by frame, each matching feature point between frames and the position information of each matching feature point on the corresponding video frame for the video to be processed.

[0011] The reconstruction unit is configured to perform the following reconstruction operation based on the detected video frames during the frame-by-frame detection process of the feature processing unit, and the feature processing unit is configured to continue processing undetected video frames:

[0012] The initial reconstruction operation is performed in the following manner: based on the number of each matching feature point and the inter-frame displacement, a first keyframe and a second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame.

[0013] After the initial reconstruction operation is completed, incremental reconstruction operations are iteratively performed in the following manner until the video processing is completed: based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed on each incremental video frame; wherein, in subsequent iterations other than the first round of incremental reconstruction operations, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

[0014] Optionally, the feature processing unit is specifically used for:

[0015] For each video frame in the video, perform the following operations in sequence:

[0016] If the number of feature points in the preceding video frame adjacent to the video frame is greater than a first number threshold, then optical flow tracing is used to determine each matching feature point between the video frame and the preceding video frame, as well as the position information of each matching feature point on the video frame and the preceding video frame.

[0017] If the number of feature points in the preceding video frame adjacent to the video frame is not greater than a first number threshold, then each feature point on the video frame is extracted, and each matching feature point between the video frame and the preceding video frame is determined by optical flow tracing, as well as the position information of each matching feature point on the video frame and the preceding video frame.

[0018] Optionally, the feature processing unit is specifically used for:

[0019] At least one instruction set acceleration method is used to perform accelerated segment test feature extraction on the video frame to obtain each feature point contained in the video frame and the response value of each feature point;

[0020] The video frame is divided into multiple filtering grids;

[0021] For each filter grid, the feature points in the filter grid are filtered according to the response value of each feature point in the filter grid.

[0022] Optionally, after filtering the feature points in the filter grid according to the response value of each feature point in the filter grid for each filter grid, the feature processing unit is further configured to:

[0023] If the number of feature points in the video frame is greater than the second number threshold, then at least one feature point with line features is deleted based on the difference between the gradient direction angles of each feature point in the video frame.

[0024] For each feature point in the video frame, whether the feature point has line features is determined in the following way:

[0025] According to the preset range parameters, the detection area is determined with the feature point as the center point;

[0026] The gradient direction angle of each feature point in the region to be detected is determined, as well as the gradient interval to which each gradient direction angle belongs; wherein each gradient interval is obtained by dividing the range of gradient direction angle values ​​at equal intervals;

[0027] Determine the target gradient interval containing the most feature points in each gradient interval, and the reference gradient interval containing the second most feature points.

[0028] If the target gradient interval contains the center point, and the angle difference between the center point and the reference gradient interval exceeds a preset angle threshold, then the center point is determined to have line characteristics.

[0029] Optionally, when detecting each matching feature point between frames and the position information of each matching feature point on the corresponding video frame, the feature processing unit is further configured to:

[0030] For each pair of adjacent frames in the video, perform at least one of the following verification operations:

[0031] By performing random sampling consistency verification on each matching feature point in two adjacent frames, matching feature points that fail the verification are deleted.

[0032] The preceding video frame in two adjacent frames is divided into multiple verification grids. For each matching feature point in each verification grid: if the difference between the optical flow displacement of the matching feature point between two adjacent frames and the corresponding average optical flow displacement is greater than a preset difference threshold, the matching feature point is deleted. The average optical flow displacement is the average value of the optical flow displacement of each matching feature point in the verification grid to which the matching feature point belongs between two adjacent frames.

[0033] Optionally, the reconstruction unit is specifically used for:

[0034] The first frame of the video is taken as the initial i-th frame, and the second frame of the video is taken as the initial j-th frame;

[0035] For the current i-th frame and j-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold.

[0036] If the selection conditions are not met, let j = j + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame;

[0037] If the selection conditions are met, pose estimation is performed on the current i-th frame and j-th frame, and based on the pose estimation results, the latest determined i-th frame is taken as the first keyframe, and the latest determined j-th frame is taken as the second keyframe.

[0038] Optionally, the reconstruction unit is specifically used for:

[0039] For the current i-th and j-th frames, determine the matching inliers between frames based on the pose estimation results;

[0040] If the total number of all matched points is greater than the fourth number threshold, then the current i-th frame is taken as the first keyframe and the current j-th frame is taken as the second keyframe.

[0041] If the total number of the matched points is not greater than the fourth quantity threshold and the total number of the matched feature points is not greater than the fifth quantity threshold, then let i = i + 1, and return to the step of analyzing whether the number of the matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame.

[0042] If the total number of all matched intrapoints is not greater than the fourth quantity threshold, and the total number of all matched feature points is greater than the fifth quantity threshold, then the preset displacement threshold is increased, j = j + 1, and the process of analyzing whether the number of all matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame is returned.

[0043] Optionally, the reconstruction unit is specifically used for:

[0044] Take the current second keyframe as the j-th frame, and take the first frame after the second keyframe as the initial k-th frame;

[0045] For the j-th frame and the current k-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold.

[0046] If the selection conditions are not met, let k = k + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the j-th frame and the current k-th frame;

[0047] If the selection criteria are met, the newly determined k-th frame will be used as the third keyframe.

[0048] Optionally, the reconstruction unit is further configured to:

[0049] The pose information corresponding to each initial video frame within the range from the first keyframe to the second keyframe is determined in the following manner:

[0050] Based on the camera intrinsic parameters and the position information of each matching feature point between the first keyframe and the second keyframe, the relative pose between the first keyframe and the second keyframe is determined.

[0051] Based on the relative pose, determine the pose information corresponding to the first keyframe and the second keyframe when the first keyframe is used as the reference coordinate system;

[0052] Based on the pose information corresponding to the first keyframe and the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to other initial video frames between the first keyframe and the second keyframe is determined frame by frame.

[0053] Furthermore, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined in the following manner:

[0054] Based on the pose information corresponding to the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined frame by frame.

[0055] Optionally, the reconstruction unit is specifically used for:

[0056] Based on the pose information corresponding to the first keyframe and the second keyframe, and combined with the position information of each matching feature point between the first keyframe and the second keyframe, two-frame triangulation processing is performed on each matching feature point between the first keyframe and the second keyframe.

[0057] For other initial video frames between the first keyframe and the second keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other initial video frames and their previous video frames, and combined with the position information of each matching feature point between the other initial video frames and their previous video frames, two-frame triangulation processing is performed on each matching feature point between the other initial video frames and their previous video frames.

[0058] The 3D point cloud obtained through two frames of triangulation during the initial reconstruction operation will be used as the initial 3D reconstruction result.

[0059] Optionally, the reconstruction unit is further configured to:

[0060] Perform at least one of the following optimization operations on the initial 3D reconstruction results:

[0061] Based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, and combined with the position information of each matching feature point between each initial video frame, multi-frame triangulation processing is performed on each matching feature point between each initial video frame; the three-dimensional point cloud obtained through multi-frame triangulation processing is added to the initial three-dimensional reconstruction result to optimize the initial three-dimensional reconstruction result.

[0062] The current initial 3D reconstruction results are optimized using global bundle adjustment (BA).

[0063] Optionally, if the first keyframe is not the first frame in the video, then during the initial reconstruction operation, the reconstruction unit is further configured to:

[0064] For the other video frames before the first keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other video frames and the next video frame, and combined with the position information of each matching feature point between the other video frames and the next video frame, two-frame triangulation processing is performed on each matching feature point between the other video frames and the next video frame.

[0065] The 3D point cloud obtained through two frames of triangulation is added to the current initial 3D reconstruction result to optimize the initial 3D reconstruction result.

[0066] Optionally, the reconstruction unit is specifically used for:

[0067] For each incremental video frame within the range from the second keyframe to the third keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the incremental video frame and its previous video frame, and combined with the position information of each matching feature point between the incremental video frame and its previous video frame, two-frame triangulation processing is performed on each matching feature point between the incremental video frame and its previous video frame.

[0068] Based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, and combined with the position information of each matching feature point between each incremental video frame, multi-frame triangulation is performed on each matching feature point between each incremental video frame.

[0069] The 3D point cloud obtained through two-frame triangulation and multi-frame triangulation during this round of incremental reconstruction will be used as the result of this round of incremental 3D reconstruction.

[0070] Incremental BA optimization was performed on the incremental 3D reconstruction results of this round.

[0071] Optionally, after obtaining the 3D reconstruction result containing the 3D point cloud, the reconstruction unit is further configured to:

[0072] The following plane detection operation is performed iteratively until a preset number of iterations is reached, or until no plane is detected in the latest round of plane detection operation:

[0073] From the 3D point cloud, select multiple 3D points that are currently not assigned to a plane. For each 3D point, perform the following operations: fit a candidate plane based on a preset number of nearest neighbors of the 3D point in the 3D point cloud; if the number of points in the candidate plane is higher than a preset threshold for the number of points in the plane, determine the average distance between the points in the plane and the candidate plane; wherein, the points in the plane are the 3D points among the currently unassigned 3D points that are less than a preset distance threshold from the candidate plane.

[0074] Compare the various average distances determined in this round of planar detection operations to determine the minimum average distance;

[0075] If the determined minimum average distance is not less than the preset error threshold, then it is determined that the plane cannot be detected in this round of plane detection operation;

[0076] If the determined minimum average distance is less than the preset error threshold, the target plane is obtained by cleaning up the in-plane points of the target candidate plane corresponding to the minimum average distance.

[0077] Optionally, the reconstruction unit is specifically used for:

[0078] Project the points in each plane corresponding to the target candidate plane onto the target candidate plane to obtain the corresponding two-dimensional points;

[0079] Based on the distance between two-dimensional points, the target candidate plane is divided into two sub-planes;

[0080] Determine the average distance between a two-dimensional point in each subplane and the center of the subplane, and the center distance between the centers of two subplanes;

[0081] If the center distance is less than the target distance threshold, then the target candidate plane is determined as the target plane; the target distance threshold is positively correlated with the average distance corresponding to the larger of the two sub-planes;

[0082] If the center distance is not less than the target distance threshold, then the larger plane is determined to be the target plane.

[0083] Optionally, during the incremental reconstruction operation, after iteratively performing the plane detection operation, the reconstruction unit is further configured to:

[0084] For each target plane that has been obtained, the target planes that meet the preset plane merging conditions are merged based on the normal vector, outer contour polygon, and outer contour polygon after mutual projection of each pair of target planes.

[0085] Optionally, after the initial reconstruction operation and the incremental reconstruction operation are completed, the reconstruction unit is further configured to:

[0086] Perform global BA optimization on the currently obtained 3D reconstruction results.

[0087] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the above-described video-based 3D reconstruction methods.

[0088] This application provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the video-based 3D reconstruction methods described above.

[0089] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the video-based 3D reconstruction methods described above.

[0090] The beneficial effects of this application are as follows:

[0091] This application provides a video-based 3D reconstruction method, apparatus, electronic device, and storage medium. Specifically, this application performs frame-by-frame feature point matching to determine the association information of feature points between frames and the position information of each matched feature point in consecutive frames, providing stable and effective inter-frame feature point tracking pairs for subsequent 3D reconstruction. In this process, after feature point matching is performed on some frames in the video, reconstruction operations can begin based on the detected video frames, while continuing frame-by-frame detection on subsequent undetected video frames.

[0092] The reconstruction operation in this paper specifically includes initial reconstruction and incremental reconstruction. In the initial reconstruction, keyframes are adaptively selected based on the number and displacement of matching feature points, thereby performing initial 3D reconstruction on each initial video frame within the range of the first to the second keyframe. The displacement reflects the motion trajectory of the matching feature points. That is, when the number of matching feature points and the length of the motion trajectory reach a certain condition, the initial reconstruction begins. By adding temporal constraints, the continuous and stable poses and spatial 3D points of some frames can be quickly recovered and the 3D scene can be reconstructed during the initial reconstruction process, effectively reducing the waiting time of objects on mobile devices and achieving faster interaction. Furthermore, after the initial reconstruction is completed, several incremental reconstructions are iteratively executed. For other video frames after the second keyframe, keyframes are also adaptively selected based on the number and displacement of matching feature points, thereby performing 3D reconstruction on each incremental video frame within the range of the second to the third keyframe, gradually optimizing the reconstruction results, improving reconstruction accuracy, and enhancing the final effect of AR effects.

[0093] In summary, this application, through feature matching and multi-threaded incremental optimization, can respond promptly to videos awaiting processing, effectively reducing object waiting time on mobile devices and achieving more accurate and faster interaction.

[0094] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0095] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0096] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0097] Figure 2 This is a schematic diagram of an AR special effect text in an embodiment of this application;

[0098] Figure 3 This is a schematic diagram of another AR special effect text in the embodiments of this application;

[0099] Figure 4 This is a flowchart illustrating an implementation of a video-based 3D reconstruction method in this application.

[0100] Figure 5 This is a schematic diagram of a line feature in an embodiment of this application;

[0101] Figure 6 This is a schematic diagram of a region to be detected in an embodiment of this application;

[0102] Figure 7 This is a schematic diagram of a gradient histogram in an embodiment of this application;

[0103] Figure 8 This is a schematic diagram of a grid partitioning scheme in an embodiment of this application;

[0104] Figure 9 This is a logic block diagram of a feature matching module in one embodiment of this application;

[0105] Figure 10 This is a flowchart illustrating the implementation of a keyframe selection method in an embodiment of this application.

[0106] Figure 11 This is a schematic diagram of an in-plane point cleanup embodiment of this application;

[0107] Figure 12 This is a logic block diagram of an initial reconstruction method in an embodiment of this application;

[0108] Figure 13This is a flowchart illustrating another keyframe selection method in this application.

[0109] Figure 14 This is a schematic diagram of an incremental BA optimization term in an embodiment of this application;

[0110] Figure 15 This is a schematic diagram of a planar merging embodiment in this application;

[0111] Figure 16 This is a logic block diagram of an incremental reconstruction method according to an embodiment of this application;

[0112] Figure 17 This is a logic block diagram of a video 3D reconstruction method according to an embodiment of this application;

[0113] Figure 18 This is a schematic diagram of the composition structure of a video-based 3D reconstruction device according to an embodiment of this application;

[0114] Figure 19 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0115] Figure 20 This is a schematic diagram of the hardware structure of another electronic device using an embodiment of this application. Detailed Implementation

[0116] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0117] The following describes some of the concepts involved in the embodiments of this application.

[0118] Feature points and matching feature points: Feature points are points in an image that possess unique properties and exhibit high local invariance, meaning they can still be identified under different viewpoints and lighting conditions. For the video to be processed in this application, several feature points can be extracted from each video frame. Matching feature points are feature points found in two or more images that have similar or identical visual appearances and positional relationships. These feature points typically maintain consistent properties across different images, even if the images were taken at different times and from different viewpoints. For the video to be processed in this application, for example, if there is a moving object in the video, then the same feature points of the object at different positions in consecutive video frames can be considered matching feature points. Matching feature points play a role in correlation and tracking across multiple video frames.

[0119] Feature point response value: Represents the intensity or salience of the feature point, which is determined by the contrast between the surrounding pixels and the center pixel value. Generally, the greater the contrast, the higher the response value and the more salience of the feature point.

[0120] 3D reconstruction is the process of establishing a mathematical model of a 3D object suitable for computer representation and processing. It is the foundation for processing, manipulating, and analyzing the properties of the object in a computer environment, and a key technology for creating virtual reality representations of the objective world within a computer. In this application's embodiments, video-based 3D reconstruction is the process of recovering a 3D scene from a series of consecutive video frames.

[0121] Initial reconstruction and incremental reconstruction are the two stages of 3D reconstruction in this embodiment. Initial reconstruction refers to the process of building a 3D model from scratch, calculating the 3D structure of the scene based on a series of 2D image inputs. Incremental reconstruction, on the other hand, updates and improves the existing 3D model using new image data to gradually refine and expand the model. These two stages work together to gradually build a complete 3D scene model, providing a reliable foundation for AR placement effects.

[0122] Instruction Set Acceleration (ISA) refers to techniques that optimize and accelerate specific types of computational tasks by utilizing the specialized hardware capabilities of a computer processor. Modern Central Processing Units (CPUs), in addition to the basic Arithmetic Logic Unit (ALU) and control unit, contain extra hardware circuitry capable of executing multiple identical or similar operations in parallel, thereby significantly increasing processing speed. This technique is typically implemented by extending the processor's Instruction Set Architecture (ISA), adding instructions specifically designed to accelerate certain types of operations.

[0123] AR (Augmented Reality) is a technology that cleverly integrates virtual information with the real world. By using various technologies such as multimedia, 3D modeling, real-time detection and registration, intelligent interaction, and sensing, virtual objects or information (such as text, images, 3D models, music, videos, virtual objects, etc.) are superimposed on real-world scenes to create an augmented reality environment. This allows objects to see both real-world and virtual-world content simultaneously, thus achieving "augmentation" of the real world.

[0124] Optical flow displacement is a form of inter-frame displacement. It describes the direction and magnitude of movement of pixels or feature points between adjacent frames in an image sequence, and is one of the important metrics for quantifying inter-frame changes. In computer vision and image processing, optical flow displacement is commonly used to analyze the motion of objects or scenes in video sequences.

[0125] Triangulation: This technique is used to determine the three-dimensional position of an object by calculating the intersection points between at least two different viewing angles. This method relies on trigonometric functions, hence the name "triangulation." In this embodiment, triangulation is primarily used to estimate the three-dimensional positions of feature points from video observation data, thereby gradually building a map of the entire environment. Specifically, two-frame triangulation and multi-frame triangulation are employed.

[0126] Two-frame triangulation refers to triangulating feature points extracted from two frames to estimate their positions in 3D space. In this process, matching feature points from each pair of frames are used to construct a triangle, thus determining the spatial coordinates of the feature points. Multi-frame triangulation involves extracting feature points from more than two frames of data and performing triangulation to estimate their 3D positions. This method can improve the accuracy and robustness of the estimation, especially in the presence of noise or occlusion.

[0127] Pose information: The camera's rotation matrix R and translation vector t relative to the world coordinate system, with 6 degrees of freedom (6-DoF): 3 degrees of rotation and 3 degrees of translation. For video, each frame is captured using the camera's pose information. Specifically, the pixels in each frame are the result of points in the 3D world projected onto the 2D image plane after the camera's pose transformation. In this process, rotation and translation determine the position of the 3D points in the image. By analyzing the feature points in the video frames and their correspondences between different frames, the camera's pose changes can be estimated. The pose information corresponding to a certain video frame in this paper refers to the pose information of the camera when capturing that video frame.

[0128] Projection constraints between 3D and 2D points refer to the mathematical relationship of how a point in 3D space is projected onto a 2D image plane. The 3D-2D Perspective-n-Point (PnP) problem in this application aims to determine the pose of a camera relative to a known set of 3D points. Given a set of points in 3D world coordinates and their 2D projections onto the image plane, the goal of the PnP algorithm is to find the camera's rotation and translation parameters such that, after projection onto the camera, these 3D points match the observed 2D points as closely as possible.

[0129] The design concept of the embodiments of this application is briefly introduced below:

[0130] With the development of computer vision, technologies such as 3D reconstruction, AR, autonomous driving, and Simultaneous Localization and Mapping (SLAM) are gradually entering our daily lives. Among them, video-based 3D reconstruction has always been one of the most important challenges in the field of computer vision. Video-based 3D reconstruction methods can recover the structure of a 3D scene from a set of 2D images (such as consecutive video frames) and are applicable to a variety of fields.

[0131] However, most current 3D reconstruction solutions for videos are based on PCs and the cloud, requiring high computing power. For example, one open-source PC software uses Scale-Invariant Feature Transform (SIFT) for feature matching and has a GPU-accelerated version. However, it processes pose optimization frame by frame, resulting in slow reconstruction speed. The average reconstruction time for a video of tens of seconds is more than 10 minutes, and the reconstruction failure rate is also high. Therefore, it is difficult to implement on mobile devices. Furthermore, there is currently no planar detection solution to provide complete 3D spatial effects (such as text effects).

[0132] In view of this, this application proposes a video-based 3D reconstruction method, apparatus, electronic device, and storage medium. Specifically, this application performs feature point matching frame by frame to determine the association information of feature points between frames and the position information of each matched feature point in consecutive frames, which is used to provide stable and effective inter-frame feature point tracking pairs for subsequent 3D reconstruction. In this process, after performing feature point matching on some frames in the video, the reconstruction operation can be started based on the detected video frames, while continuing to detect subsequent undetected video frames frame by frame.

[0133] In the initial reconstruction operation, keyframes are adaptively selected based on the number and displacement of matching feature points, thereby performing initial 3D reconstruction on each initial video frame within the range of the first to the second keyframe. The displacement reflects the motion trajectory of the matching feature points. That is, when the number of matching feature points and the length of the motion trajectory reach a certain condition, the initial reconstruction begins. By adding temporal constraints, the continuous and stable poses and spatial 3D points of some frames can be quickly recovered and the 3D scene can be reconstructed during the initial reconstruction process, effectively reducing the waiting time of objects on mobile devices and achieving faster interaction. Furthermore, after the initial reconstruction is completed, several incremental reconstructions are iteratively executed. For other video frames after the second keyframe, keyframes are also adaptively selected based on the number and displacement of matching feature points, thereby performing 3D reconstruction on each incremental video frame within the range of the second to the third keyframe, gradually optimizing the reconstruction results, improving reconstruction accuracy, and enhancing the final effect of AR effects.

[0134] In summary, this application, through feature matching and multi-threaded incremental optimization, can respond promptly to videos awaiting processing, effectively reducing object waiting time on mobile devices and achieving more accurate and faster interaction. Furthermore, this application proposes a fast spatial multi-plane detection algorithm to quickly support interactions such as object placement in spatial planes.

[0135] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0136] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes a terminal device 110 and a server 120.

[0137] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client related to video 3D reconstruction installed. This client can be software (e.g., browsers, video software, game software, navigation software), or a webpage, mini-program, etc. The server 120 is the backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for video 3D reconstruction; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0138] It should be noted that the video-based 3D reconstruction method in the embodiments of this application can be executed by an electronic device, which can be a terminal device 110 or a server 120. That is, the method can be executed by the terminal device 110 or the server 120 alone, or by both the terminal device 110 and the server 120. For example, when executed by both the terminal device 110 and the server 120, a 3D reconstruction-related client can be installed on the terminal device 110. The user can input the video to be processed based on the client. Then, for the video to be processed, the terminal device 110 can detect the matching feature points between frames and the position information of each matching feature point on the corresponding video frame frame by frame. At the same time, during the frame-by-frame detection process using the matching thread by the terminal device 110, the server 120 can perform the following reconstruction operations (including initial reconstruction operation and incremental reconstruction operation) based on the detected video frames. Meanwhile, the terminal device 110 continues to process the undetected video frames in parallel.

[0139] The initial reconstruction operation is performed as follows: Based on the number of matching feature points and the inter-frame displacement, the first keyframe and the second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame.

[0140] After the initial reconstruction operation is completed, the incremental reconstruction operation is iteratively executed in the following manner until the video processing is completed: Based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed for each incremental video frame; wherein, in the subsequent iteration process except for the first round of incremental reconstruction operation, the second keyframe in each round is the third keyframe selected in the adjacent previous round.

[0141] When performing the initial reconstruction operation, the initial 3D reconstruction result is obtained and then fed back to the terminal device 120, which uses this result to display AR effects. Furthermore, when iteratively performing incremental reconstruction operations, the result of each incremental reconstruction can also be fed back to the terminal device 120 in real time, which uses this result to further display AR effects.

[0142] AR technology has a wide range of applications, including education, gaming, industrial design, and healthcare. For example, in education, AR technology can be used to create virtual laboratories and virtual field trips; in gaming, it can be used to create augmented reality games, allowing players to interact with virtual objects in the real world; in industrial design, it can be used for product design and prototyping; and in healthcare, it can be used for surgical simulations and medical education. AR functionality is widely used in short videos, games, and UI interactions, with AR virtual object placement being adopted as a new interaction method in many applications.

[0143] Specifically, the video-based 3D reconstruction method provided in this application can be applied in AR scenarios on mobile devices where virtual objects are placed. AR virtual object placement refers to placing virtual objects in specific locations in the real world within an AR environment. This technology uses the cameras and sensors of AR devices (such as smartphones, tablets, or AR glasses) to identify objects and locations in the real world and overlay virtual objects onto these objects.

[0144] For example, in an AR game, a player can place a QR code marker on a flat surface in the real world and then scan the marker with an AR device. The system will then display a virtual character or prop at that location. As another example, in a furniture shopping app, users can use AR devices to place virtual furniture in their real-world home space to see how it looks. Furthermore, users can use gestures to place virtual furniture in the real space and adjust its position and angle.

[0145] Plane detection is a crucial capability for placing AR virtual objects. It refers to the technology of identifying and locating planes in the real world and placing virtual objects on these planes. Plane detection is a key step in integrating virtual objects with the real world. The accuracy and stability of plane detection are critical to the quality of AR applications. This application also proposes a plane detection method with high accuracy and stability, supporting plane detection during both initial and incremental reconstruction processes to facilitate subsequent AR virtual object placement.

[0146] In summary, the method proposed in this application can be used on various mobile electronic devices (but not limited to mobile devices), such as mobile phones, tablets, and laptops. It can respond to video input from objects in a timely manner, perform fast and efficient spatial iterative optimization reconstruction and planar recognition on mobile devices, and quickly reconstruct 3D scenes from 2D images, thereby supporting the realization of AR placement effects such as object placement and wall advertising text.

[0147] like Figure 2 The diagram shown illustrates an AR special effects text example from an embodiment of this application. In a short video application, users can place a target object in real space to view related special effects text. Users can intuitively see the integration of the virtual object and its special effects text with the surrounding environment. Users can also adjust the placement based on the prompt "Click here to reset," thereby achieving a more natural and satisfying AR experience.

[0148] like Figure 3 As shown, this is a schematic diagram of another AR special effects text in an embodiment of this application. The viewer can see virtual advertising text superimposed on a real wall. This text not only seamlessly integrates with the real environment but may also possess dynamic effects or interactive functions, providing a more vivid and engaging experience.

[0149] Of course, the above are just simple examples. The video-based 3D reconstruction method provided in this application can be applied not only to AR, Virtual Reality (VR) and other scenarios to construct realistic virtual environments through 3D reconstruction technology, but also to other fields. A few examples are listed below:

[0150] Industrial Design and Manufacturing: Using 3D reconstruction technology during the product design phase helps engineers better understand the product's structure and facilitates quality control and defect detection during manufacturing. Architecture and Urban Modeling: Reconstructing 3D models of buildings or cities using videos captured by drones or handheld devices can be applied to urban planning, architectural design, and other fields. Film Special Effects: In film production, 3D reconstruction of scenes or actors' movements can create realistic visual effects and animations. Medical Imaging: In the medical field, video 3D reconstruction can be used to create 3D models of patient organs, assisting in surgical planning and treatment decisions. Game Development: Using 3D reconstruction technology in games can create realistic environments or character models, enhancing the game's realism and immersion. Digitization of Cultural Heritage: Capturing details of artifacts or historical sites through video and then performing 3D reconstruction can be used for the preservation and research of historical and cultural heritage; and so on.

[0151] It should be noted that the above examples are just simple illustrations. Any video-based 3D reconstruction scenario is applicable to the embodiments of this application, and will not be repeated here.

[0152] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0153] In one alternative implementation, the communication network is a wired network or a wireless network.

[0154] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0155] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the video-based 3D reconstruction method of this application, the video-related data involved can be stored on the blockchain, such as video, feature point related information, pose information, 3D reconstruction results, etc.

[0156] Furthermore, the embodiments of this application can be applied to various fields, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0157] It should be emphasized that the specific embodiments of this application involve object-related data, such as the videos input by the object as listed above. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the object is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0158] The following describes the video-based 3D reconstruction method provided by the exemplary embodiments of this application in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0159] See Figure 4 The diagram shown is a flowchart of an implementation of a video-based 3D reconstruction method provided in this application. Taking a server as the execution entity, the specific implementation process of this method is as follows: S41-S42:

[0160] S41: For the video to be processed, detect each matching feature point between frames and the position information of each matching feature point on the corresponding video frame; during the frame-by-frame detection process, perform the reconstruction operation in S42 based on the detected video frames, and continue to process the undetected video frames.

[0161] The videos to be processed can be any type of video, including those of any length, not limited to short or medium-length videos, and can even be live video streams. Classified by video type, they can include game videos, drone aerial videos, etc. For example, game videos can be used to create 3D models of game environments; drone aerial videos can be used for updating geographic information systems, urban planning, and disaster assessment. Other examples include sports event videos, educational videos, and vehicle driving videos. Sports event videos can be used to reconstruct athletes' movements and competition venues for technical analysis and training guidance; educational videos can be used to create teaching resources, such as virtual laboratories or historical scenes, to enhance the learning experience; and videos of vehicles in motion can be used to build 3D models of roads and their surroundings for testing and validating autonomous driving systems, and so on.

[0162] It should be noted that the videos listed above are just simple examples, and any kind of video is applicable to the embodiments of this application, and will not be described in detail here.

[0163] For the video to be processed, before feature matching, the video can be compressed based on computing power to improve processing speed. One possible compression method is to compress the video to a preset video resolution.

[0164] Specifically, the preset video resolution can specify the number of pixels constituting the video image. In this embodiment, at least one of the number of pixels in the width of the video image and the number of pixels in the height of the video image can be specified.

[0165] For example, a video with an original resolution of 1080p (i.e., 1920×1080) means that each frame of the video consists of 1920 horizontal pixels and 1080 vertical pixels; another example is a video with an original resolution of 720p (i.e., 1280×720), which means that each frame of the video consists of 1280 horizontal pixels and 720 vertical pixels; yet another example is a video with an original resolution of 480p (i.e., 854×480), which means that each frame of the video consists of 854 horizontal pixels and 480 vertical pixels; and so on.

[0166] To ensure both quality and speed, the default video resolution can be set to 640 pixels on the longer side (the side with the greater width and height). Therefore, for videos with an original resolution higher than 720p, the longer side can be adjusted to 640 pixels.

[0167] For example, for landscape videos, the width is adjusted to 640 pixels, and the height is scaled proportionally; for portrait videos, the height is adjusted to 640 pixels, and the width is scaled proportionally.

[0168] In the above implementation, by compressing the video, the computational burden can be reduced, ensuring better processing results and faster processing speed with limited computing resources.

[0169] In this embodiment of the application, after the video is compressed, the feature matching part can be entered. This part is used for feature point extraction and tracking, and outputs more robust and stable inter-frame feature point association information and position information of each feature point in consecutive frames through a variety of efficient strategies.

[0170] Specifically, this part can be implemented based on a matching thread (which can also be understood as a feature matching module). The specific implementation process of this part is divided into two steps: feature point extraction and feature point tracking. In the embodiments of this application, feature point extraction and feature point tracking can be executed in parallel. For example, while performing feature point extraction on subsequent video frames, feature point tracking can be performed on previous video frames, thereby effectively managing computing resources and improving processing speed while maintaining tracking accuracy.

[0171] Furthermore, this application takes into account that feature point extraction is a computationally intensive task, especially in high-definition video, and that there is usually a high degree of similarity between adjacent frames in video sequences (i.e., feature points often have a certain degree of stability between consecutive frames), especially in slow-motion or slightly moving scenes. This means that in the absence of drastic motion or environmental changes, feature points from the previous frame still exist in the current frame and can be tracked. Directly tracking these points can maintain the consistency and continuity of feature points.

[0172] Therefore, this application proposes a dynamic balancing mechanism that intelligently decides whether to perform feature point extraction or direct tracking based on a comparison of the number of feature points in the current frame and the previous frame. This optimizes computational efficiency and resource consumption while ensuring the robustness and accuracy of the algorithm. Specifically, before extracting feature points from a video frame, the number of feature points in the previous frame is checked. If a sufficient number of feature points have been extracted from the previous frame, then the video frame is directly tracked. Accordingly, an optional implementation of S41 is as follows:

[0173] For each video frame, perform the following operations sequentially:

[0174] For a video frame, if the number of feature points in the preceding video frame is greater than a first threshold, then optical flow tracing is used to determine each matching feature point between the video frame and its preceding video frame, as well as the position information of each matching feature point in the video frame and its preceding video frame.

[0175] If the number of feature points in the preceding video frame adjacent to the current video frame is not greater than the first number threshold, then each feature point on the current video frame is extracted, and each matching feature point between the current video frame and its preceding video frame is determined by optical flow tracing, as well as the position information of each matching feature point on the current video frame and its preceding video frame.

[0176] In other words, for a video frame, if the video frame is the first frame in the video, it means that there is no previous video frame, and feature point extraction is performed directly on the video frame; if the video frame is not the first frame in the video, it is further analyzed whether the number of feature points in the previous video frame is greater than a first number threshold; if it is greater than the first number threshold, feature point extraction is no longer performed on the video frame, and the feature points of the previous frame are tracked directly; if it is not greater than the first number threshold, feature point extraction and feature point tracking are performed on the video frame.

[0177] Repeat the above steps for each video frame until the video ends.

[0178] The first quantity threshold is a pre-set threshold related to the number of feature points. It can be flexibly set according to factors such as experience, actual needs, computing resources, video content itself, and accuracy requirements of 3D reconstruction. This paper does not impose specific limitations on it.

[0179] Assume the first quantity threshold is denoted as T. detect A value of 1000 indicates that if the number of feature points in the preceding video frame exceeds 1000, feature point extraction for that video frame can be skipped. The following is a simple example illustrating the feature matching part of the video:

[0180] For example, feature point extraction is performed on the first frame of the video, yielding 1200 feature points. For the second frame, since the first frame has a sufficient number of feature points, feature point tracking is performed on the first frame, resulting in 1150 feature points. These 1150 feature points are matching feature points between the second and first frames. For the third frame, feature point tracking is performed on the second frame, yielding 950 feature points. These 950 feature points are matching feature points between the third and second frames, and also matching feature points between the third and first frames, obtained through indirect matching based on the second frame. For instance, for feature point A in the first frame, a matching feature point A' is tracked in the second frame, and a matching feature point A” is tracked based on A' in the third frame. Therefore, A and A' are matching feature points between the first and second frames, A' and A” are matching feature points between the second and third frames, and A and A” are also matching feature points between the first and third frames.

[0181] For the fourth frame, since the number of feature points in the third frame has dropped to 950, which is below the threshold Tdetect, feature point extraction is performed on the fourth frame. Assuming 1050 new feature points are extracted, feature point tracking is performed on the third frame to determine the matching feature points between the third and fourth frames. For the fifth frame, since the number of feature points in the fourth frame is sufficient, feature point tracking is performed on the fourth frame, yielding 1000 feature points. These 1000 feature points are the matching feature points between the fourth and fifth frames. Furthermore, since new feature points were extracted in the fourth frame, these 1000 feature points do not belong to the matching feature points of other frames before the fourth frame. If necessary, they can be re-extracted and tracked.

[0182] For subsequent frames, the above steps are repeated, and the decision to perform feature point extraction is dynamically made based on the number of feature points.

[0183] Through the above process, the association information and pose information of feature points between two or more frames in the video can be obtained, that is, the matching feature points between frames and their position information in the corresponding video frames obtained in S41.

[0184] In the embodiments of this application, these feature points that have a matching relationship between frames can be called matching feature points, and the position of each matching feature point may be different in different video frames.

[0185] For example, for two consecutive video frames (which can be simply referred to as frames): Frame1 and Frame2, after feature matching, the following matching relationship is obtained: A1 matches B1, A2 matches B2, and A3 matches B3. Then the position of feature point A1 in Frame1 (100, 150) matches the position of feature point B1 in Frame2 (105, 155), and so on.

[0186] For example, considering three consecutive frames: Frame1, Frame2, and Frame3, assuming that feature matching yields the following matching relationships: A1 matches B1, A2 matches B2, B1 matches C1, B2 matches C2, and B3 matches C3, then the position of feature point A1 in Frame1 (100, 150) matches the position of feature point B1 in Frame2 (105, 155), and the position of feature point B1 in Frame2 (105, 155) matches the position of feature point C1 in Frame3 (110, 160). Similarly, A2 and B2, as well as B2 and C2, also match. Furthermore, feature point B3 appears in Frame2 and matches feature point C3 in Frame3.

[0187] In the above embodiments, by reducing unnecessary feature point extraction steps, the computational burden can be significantly reduced, and the processing speed of video frames can be accelerated. Furthermore, when the number of feature points falls below a threshold, feature point extraction is retried, thereby ensuring sufficient feature points for subsequent feature point tracking and maintaining high tracking accuracy. Moreover, this application is particularly applicable to resource-constrained mobile electronic devices such as mobile phones. On mobile electronic devices, based on the above embodiments, unnecessary computation can be reduced, lowering the complexity of subsequent processing and thus improving processing speed, minimizing latency while ensuring performance.

[0188] The above describes the feature point extraction and feature point tracking process. The following section will first describe the specific implementation method for feature point extraction:

[0189] For a video frame, in one optional implementation, at least one instruction set acceleration method can be used to perform accelerated segment test feature extraction on the video frame to obtain each feature point contained in the video frame and the response value of each feature point.

[0190] Specifically, the Features from Accelerated Segment Test (FAST) algorithm mainly detects whether there are regions with significant brightness changes around a pixel in order to determine whether the pixel is a feature point.

[0191] In this application embodiment, the extended instruction set includes, but is not limited to, some or all of the following:

[0192] Streaming SIMD Extensions (SSE), Advanced Vector Extensions (AVX), AVX-512, Media Processing Engine (NEON), Fused Multiply-Add (FMA), etc.

[0193] In this embodiment, FAST feature extraction is performed using one or more instruction sets. These instruction sets provide vectorized operations that can process multiple data in parallel, thereby significantly improving the speed of FAST feature detection and making it more suitable for real-time applications and large-scale data processing.

[0194] For example, in feature point extraction, SIMD instructions can be used to process pixels of video frames in parallel, thereby greatly accelerating the detection speed of feature points. In the FAST algorithm, a central pixel is considered a feature point if a sufficient number of consecutive pixels (e.g., at least 7 out of 9) in its surrounding annular region are brighter or darker than the central pixel by more than a certain threshold.

[0195] Assuming that for each video frame, FAST feature extraction is used to accelerate the instruction set, n feature points are obtained in the whole image. The response value of each feature point is si, where i takes the value from 1 to n. The response value of each feature point reflects the strength or significance of that feature point.

[0196] Of course, besides the FAST algorithm, other feature point extraction algorithms can also be used, such as SIFT, SURF (Speeded UpRobust Features), ORB (Oriented FAST and Rotated BRIEF), and Harris corner detection. SIFT detects feature points by finding extrema in spatial pyramids at different scales and calculates the response value for each candidate feature point; SURF uses a fast approximation algorithm based on integral images to detect feature points and calculates response values ​​to evaluate the stability of the feature points; ORB uses the FAST algorithm to detect feature points and uses the BRIEF descriptor to calculate the response values ​​of the feature points; the Harris corner detection algorithm can detect corners in images, which typically have good localization stability and directionality, and so on; this paper does not specifically limit these algorithms.

[0197] For each video frame, after obtaining all feature points and their response values, in order to ensure that the feature points are evenly distributed, feature points can be further filtered based on these response values ​​to remove those with low response values ​​or that may be unstable.

[0198] In this embodiment of the application, a grid strategy can be used to filter feature points. One possible implementation is to divide the video frame into multiple filtering grids; for each filtering grid, the feature points in the filtering grid are filtered according to the response value of each feature point in the filtering grid.

[0199] Specifically, for each video frame, the image (hereinafter referred to as the image or video frame) can be divided into multiple grids, referred to here as filter grids. The size of each filter grid is determined by a preset pixel threshold. For example, the image can be divided into a 2×2 filter grid, with each filter grid containing 4 pixels. Another example is a 5×5 filter grid, with each filter grid containing 25 pixels. Of course, this pixel threshold can also be other sizes, and can be flexibly set based on experience, actual needs, computing resources, the video content itself, and the accuracy requirements of 3D reconstruction, etc. This article does not impose specific limitations on this.

[0200] It should be noted that while each filtering grid contains the same number of pixels, the number of feature points in each filtering grid is uncertain because the feature points are automatically determined by the detection algorithm based on image content (such as texture, contrast, etc.).

[0201] Therefore, before filtering feature points in a filter grid based on response values, it is necessary to consider whether the number of feature points in the filter grid is sufficient. In this embodiment, the filtering at this stage specifically ensures that at least one feature point with a high response value is retained in each filter grid (except for filter grids that do not contain any feature points). For example, if this application aims to ensure that each filter grid retains only the feature point with the highest response value, then only filter grids containing two or more feature points need to be filtered. Similarly, if this application aims to ensure that each filter grid retains the two feature points with the highest response values, then only filter grids containing three or more feature points need to be filtered, and so on. Furthermore, if a filter grid contains no feature points, then that filter grid does not retain any feature points.

[0202] When retaining only the feature point with the highest response value for each filter grid, if multiple feature points in a filter grid have the same highest response value, one can be randomly selected for retention, or selected according to a certain order (such as from left to right, from top to bottom), or based on positional priority, prioritizing feature points near the grid center, or based on other properties of the feature points, such as the consistency of gradient direction, the distance between feature points, etc., selecting the feature point that best matches the other retained feature points in the grid, and so on. Of course, the same principle applies when retaining the two (or other number) feature points with the highest response values ​​for each filter grid, and this paper does not impose specific limitations on this.

[0203] In the above implementation, the instruction set acceleration can significantly improve the speed of FAST feature detection, making it more suitable for real-time applications and large-scale data processing. Furthermore, the grid strategy ensures that at least one feature point with a high response value is retained in each filtering grid (except for filtering grids that do not contain feature points themselves), thereby improving the uniformity of feature point distribution throughout the image. Simultaneously, it retains feature points with high response values ​​and filters out feature points with low response values, thus improving the accuracy and efficiency of subsequent processing.

[0204] Next, when the number of feature points in a video frame exceeds the set threshold (i.e., the second number threshold in this paper), it means that there are enough features. At this time, we can consider further removing some feature points that are not conducive to subsequent detection and tracking, such as feature points with line features (which can be referred to as line feature points), because this type of feature has simple texture and is not conducive to subsequent tracking.

[0205] One optional implementation is to perform the following operations for each video frame:

[0206] After filtering feature points using the above grid strategy, analyze whether the number of feature points in the video frame exceeds the second threshold:

[0207] If the number of feature points in the video frame is not greater than the second threshold, it means that the feature point extraction of the video frame is complete, and subsequent processing (such as feature point tracking) can continue.

[0208] If the number of feature points in the video frame is greater than the second threshold, then at least one feature point with line features in the video frame needs to be deleted based on the difference between the gradient direction angles of each feature point in the video frame, that is, one or more line feature points in the video frame need to be deleted.

[0209] Specifically, in this step, for a video frame, each detected line feature point can be deleted; or only some line feature points can be deleted until the number of remaining feature points in the video frame is equal to the second quantity threshold; or a specified number of line feature points can be filtered, etc. This article does not make specific limitations on this.

[0210] The second quantity threshold is another pre-set threshold related to the number of feature points. It can be flexibly set according to factors such as experience, actual needs, computing resources, video content itself, and accuracy requirements of 3D reconstruction. This paper does not make specific limitations on it.

[0211] like Figure 5 As shown, it is a schematic diagram of a line feature in an embodiment of this application. Wherein, Figure 5 The image provided is a partial view of a keyboard. The part marked by the dashed box is a line feature, containing multiple line feature points.

[0212] In this embodiment of the application, the purpose of this step is to remove feature points with line features from the video frame, that is, feature points with small differences between gradient direction angles, because such feature points are often associated with line features.

[0213] For such feature points, an image gradient removal method can be used. First, for each feature point, the pixel gradient direction angle in its region is calculated. Then, the gradient direction angles of the pixels around the feature point are analyzed. If the gradient direction angles of most pixels are similar, it indicates that there are line features near the feature point. Finally, if the gradient direction angles of the pixels around the feature point are small, the feature point is considered to have line features, and these feature points with line features are deleted to reduce the influence of line features in the feature points.

[0214] In this embodiment of the application, an optional method for detecting line feature points is as follows:

[0215] For each feature point, perform the following operations:

[0216] First, the detection area is determined with the feature point as the center point according to the preset range parameters.

[0217] Subsequently, the gradient direction angle of each feature point in the detection area is determined, as well as the gradient interval to which each gradient direction angle belongs; wherein, each gradient interval is obtained by dividing the range of gradient direction angle values ​​at equal intervals.

[0218] Finally, the target gradient interval containing the most feature points and the reference gradient interval containing the second most feature points are determined in each gradient interval. If the target gradient interval contains the center point and the angle difference between the target gradient interval and the reference gradient interval exceeds a preset angle threshold, then the center point is determined to have line features and can be deleted.

[0219] When there are multiple target gradient intervals and / or reference gradient intervals, it is sufficient if any one of the target gradient intervals and the reference gradient interval satisfies the above conditions. Furthermore, since a gradient interval is an angular range, the angular difference between the target gradient interval and the reference gradient interval can specifically refer to the difference between the upper and lower limits of that angular range.

[0220] In the embodiments of this application, the preset range parameters can be flexibly set according to factors such as experience, actual needs, computing resources, video content itself, and accuracy requirements of 3D reconstruction. This document does not impose specific limitations on them.

[0221] Specifically, the delineation of the detection region was performed based on the previously defined filtering grid. Assuming that each filtering grid retains only one feature point with the highest response value (for clarity, we assume there are no empty filtering grids), then:

[0222] Assuming the preset range parameter is denoted as P1, if P1 = 10, then for a certain feature point, five feature points are taken from the top, bottom, left, and right sides of it, forming a detection area containing 11×11 filtering grids, for a total of 121 feature points; or, for example, if P1 = 6, then for a certain feature point, three feature points are taken from the top, bottom, left, and right sides of it, forming a detection area containing 7×7 filtering grids, for a total of 49 feature points; and so on.

[0223] like Figure 6 As shown, this is a schematic diagram of a region to be detected in an embodiment of this application. Taking P1=6 as an example, in Figure 6 The text lists two regions to be detected. The solid box represents the region to be detected centered on feature point A, which contains 49 feature points. The dashed box represents the region to be detected centered on feature point B, which also contains 49 feature points.

[0224] Then, for each of the 49 feature points, within the small area to be detected, the gradient direction angle of each feature point is calculated one by one.

[0225] Specifically, the gradient direction angle of a pixel in an image is calculated based on the rate of change of intensity at that point in the image. The gradient direction angle is usually aligned with the direction of the greatest change in image intensity, i.e., along this direction the image brightness changes the fastest. In image processing, the gradient direction angle can be obtained by calculating the horizontal and vertical components of the image gradient, and then using the arctangent function to determine the direction.

[0226] One possible calculation method is as follows:

[0227]

[0228] Among them, angle fi That is, the gradient direction angle of a pixel, x diff and y diff These represent the pixel difference in the x and y directions of the image, respectively. This pixel difference can be determined based on its neighboring pixels (top / bottom / left / right). Using the arctan function here ensures that the quadrant of the angle is correctly determined.

[0229] For example, for feature point A, x diff This represents the pixel difference between the feature point and its adjacent feature point in the x-direction of the image; correspondingly, y... diff This represents the pixel difference between the feature point and its adjacent feature point in the y-direction of the image; for feature point B, x diff This represents the pixel difference between the feature point and its adjacent feature point in the x-direction of the image; correspondingly, y... diffThis means that the pixel difference between the feature point and its adjacent feature point in the y-direction of the image; the same applies to other feature points, which will not be elaborated here.

[0230] For each feature point within a region to be detected, its corresponding gradient direction angle can be calculated using the above method, and then the histograms corresponding to these gradient direction angles within the region to be detected can be plotted.

[0231] Specifically, the possible range of gradient direction angle values ​​can be divided into multiple gradient intervals at equal intervals according to a preset angle. The preset angle can be flexibly set according to experience or actual needs. Alternatively, the number of gradient intervals to be divided, H1, can be specified in advance, and the preset angle can be determined and divided accordingly.

[0232] When the gradient direction angle is calculated using Formula 1 above, the range of the arctangent function is (-π / 2, π / 2) radians, or equivalently, (-90°, 90°) degrees, totaling 180°. Assuming H1 = 5, the preset angle is 180° / 5 = 36°, and it is planned to be divided into 5 gradient intervals, such as:

[0233] Interval 1: (-90°, -54°]; Interval 2: (-54°, ​​-18°]; Interval 3: (-18°, 18°]; Interval 4: (18°, 54°]; Interval 5: (54°, 90°).

[0234] Of course, for easier understanding, this can also be achieved by adding 90° to the calculated angle, adjusting the range of the arctangent function to (0°, 180°), thus obtaining five gradient intervals as follows:

[0235] Interval 1: (0°, 36°]; Interval 2: (36°, 72°]; Interval 3: (72°, 108°]; Interval 4: (108°, 144°]; Interval 5: (144°, 180°).

[0236] like Figure 7 As shown, this is a schematic diagram of a gradient histogram in an embodiment of this application. Figure 7 The list shows the five intervals corresponding to the second type of interval mentioned above, namely (0°, 180°).

[0237] For a region to be detected, the gradient direction angle of each feature point in the region is calculated one by one, and the number of gradient direction angles in each interval (i.e., every 36 degrees as listed above) is counted. These results can then be grouped into a histogram, where the interval containing the most feature points is the target gradient interval, such as... Figure 7 In interval 4, the interval with the second highest number of feature points is the reference gradient interval, such as... Figure 7The angular difference between interval 2, interval 4 and interval 2 is 72.

[0238] After obtaining the gradient histogram, check whether the gradient direction angle of the center point belongs to the target gradient interval. If it does, and this target gradient interval is significantly higher than the reference gradient interval (i.e., exceeds the preset angle threshold T), then... angle If the center point is located on a distinct line, it is considered not a good feature point and cannot pass the line feature check, also known as the direction consistency check, because its direction is too affected by the surrounding lines and lacks uniqueness. Therefore, the center point can be deleted.

[0239] In summary, the above implementation means that if a feature point passes the orientation consistency test—that is, if its gradient orientation angle is not completely consistent with the dominant orientation in its neighborhood or the dominance of the dominant orientation is not obvious enough—then the feature point is considered valid and can be retained. Otherwise, the feature point is considered a line feature point and will be deleted to avoid introducing erroneous or unstable results in subsequent feature matching or description processes. This method combines feature point detection and orientation consistency evaluation, effectively utilizing the statistical properties of local gradient orientation angles to improve the quality of feature points, filtering out points that are inconsistent with their surroundings, especially those that may be located on linear structures in the image, ensuring that only those points with significant orientation diversity and stability are retained, which is beneficial to improving the accuracy and robustness of subsequent feature matching.

[0240] It should be noted that the above example assumes that there are no empty filter grids in the area to be detected. In practical applications, if empty filter grids exist, the number of feature points will be less than 49. Accordingly, the missing feature points can be ignored. In addition, in practical applications, some feature points near the image edge may not be able to be used as center points. Similarly, these feature points can be ignored. No specific restrictions are imposed here.

[0241] Of course, besides the implementation methods listed above that use image gradients to remove line features, line feature point detection methods also include: Hough transform, structure tensor, directional filter bank, machine learning, template matching, and clustering after feature point matching. Among these, the Hough transform can effectively detect a set of straight lines or curves in an image and has good robustness to noise. The structure tensor method detects edges and lines by analyzing the local directionality and intensity in the image, helping to determine the dominant direction in the image and thus find line features. The directional filter bank method uses a set of directional filters to detect strong edge responses in different directions, making it particularly suitable for detecting lines in specific directions. Machine learning methods, such as Convolutional Neural Networks (CNNs), can be trained to recognize complex line feature patterns. Template matching compares predefined templates with local regions in the image to identify similar line features. Clustering after feature point matching involves first detecting corner points or feature points in the image, and then discovering line features by matching these points and clustering them spatially. These methods each have their own characteristics and can be flexibly selected according to the application scenario.

[0242] After the above steps, feature point extraction is complete. The next step is tracking, and the specific implementation method of feature point tracking will be described below:

[0243] In the embodiments of this application, the classic optical flow tracking method can be used to track feature points and obtain effective tracking points between two frames, which are referred to as matching feature points in this paper.

[0244] Specifically, optical flow tracking is used to estimate the direction and velocity of movement of each pixel or feature point in an image sequence between adjacent frames. It assumes that pixel brightness does not change significantly in a short period (i.e., the brightness constant assumption), meaning the movement of feature points between adjacent frames is smooth and predictable. Therefore, the displacement of feature points is estimated by solving for the spatiotemporal derivative of pixel brightness, i.e., tracking the position of feature points in the next frame. Based on this method, the inter-frame feature point association information and the position information of each feature point in consecutive frames can be determined.

[0245] In this application embodiment, the optical flow tracing algorithm includes, but is not limited to, some or all of the following:

[0246] Algorithms such as Lucas-Kanade, Horn-Schunck, Farneback, and DeepFlow (a deep learning-based method) are mentioned.

[0247] In summary, the optical flow tracking results in the embodiments of this application typically include the following information: the direction and velocity of movement of each feature point (usually expressed as the magnitude of displacement). Therefore, the result of optical flow tracking is a two-dimensional vector field, where each vector represents the optical flow displacement vector of a feature point in the image. These optical flow displacement vectors together constitute the optical flow field describing the inter-frame changes of the entire image.

[0248] For a given feature point, the optical flow tracing algorithm attempts to find the corresponding position of this feature point in the current frame in the previous frame, thereby calculating the displacement between these two positions, i.e., the optical flow displacement vector (u). i v i ), where u i v represents the displacement in the horizontal direction (u). i It represents the displacement in the vertical direction (v).

[0249] For complex scenarios, optical flow vectors can be optimized through multiple iterations to improve tracking accuracy. Furthermore, considering a larger search range, pyramid structures or multi-scale methods can be used to improve computational efficiency and robustness.

[0250] Due to factors such as changes in illumination, rapid movement, and occlusion, optical flow tracking may produce incorrect matches. Therefore, to ensure the reliability of the match, outlier removal is usually required after optical flow tracking. In this application, some feature points can be removed through the following verification operation:

[0251] In one alternative implementation, for every two adjacent frames in the video, at least one of the following verification operations may be performed:

[0252] Verification Operation 1: Perform random sampling consistency verification on each matching feature point in two adjacent frames, and delete the matching feature points that fail the verification.

[0253] Specifically, Random Sampling Consensus (RANSAC) verification estimates the preliminary essential matrix by randomly sampling the minimum necessary number of matching feature point pairs, performs a consistency check on all matching feature point pairs, calculates the projection error, and considers point pairs with errors within the verification threshold as consistent point pairs, while the rest are considered as verification failures. This verification threshold is a key parameter used to identify and exclude erroneous feature points.

[0254] Furthermore, to balance efficiency and accuracy, the consistency check threshold can be dynamically adjusted based on the current number of feature points. For example, when there are many feature points, increasing the check threshold can eliminate outliers more quickly, reduce computation time, and prevent high check time consumption; conversely, if there are few feature points, the check threshold can be decreased to ensure that as many true matches as possible are retained.

[0255] In the embodiments of this application, after one or more RANSAC verifications, feature points that are incorrectly matched can be effectively identified and excluded, thereby improving the accuracy and robustness of feature matching.

[0256] Verification Operation 2: Divide the preceding video frame in two adjacent frames into multiple verification grids; for each matching feature point in each verification grid: if the difference between the optical flow displacement of the matching feature point in two adjacent frames and the corresponding average optical flow displacement is greater than the preset difference threshold, then the matching feature point is considered to have failed verification and is deleted in both adjacent frames.

[0257] The average optical flow displacement is the average optical flow displacement of each matching feature point within the verification grid to which the matching feature point belongs between two adjacent frames.

[0258] In this second verification operation, the optical flow displacement of the feature point is a vector. Therefore, when comparing the differences in optical flow displacement, at least one of the distance difference and the angle difference can be compared. Thus, this second verification operation can also be summarized as a distance and angle verification.

[0259] Specifically, a grid partitioning verification method is used, dividing the image into an N×M verification grid for verification. This verification grid is a further subdivision based on the aforementioned filtering grid; therefore, the filtering grid can be referred to as the small grid, and this verification grid as the large grid. N and M can be the same or different. For example, N×M = 2×2 means that one verification grid contains four filtering grids; another example is N×M = 3×3, which means that one verification grid contains nine filtering grids. Figure 8 As shown; for example, N×M=2×3 means that a check grid contains six filter grids.

[0260] This method divides the entire image space into multiple check grids, where feature points are considered to have similar motion characteristics. Therefore, feature points within each check grid are processed individually to evaluate their motion consistency.

[0261] Specifically, for each verification grid, the average vector (u', v') of the optical flow tracking results of the matched feature points within that grid is calculated. This average vector represents the average motion trend of the matched feature points within the verification region, denoted as the average optical flow displacement vector. If all matched feature points move in the same direction, the average optical flow displacement vector will point in that direction; if any matched feature points move in the opposite direction, the direction of the average optical flow displacement vector may be pulled towards the center, reflecting the average motion direction and velocity within the grid region. Then, the optical flow displacement vector (u', v') of each matched feature point is verified. i v i The difference in length and angle between the average optical flow displacement vector (u', v') and the average optical flow displacement vector (u', v'), where the length difference can be expressed as the Euclidean distance between the two vectors and the angle difference can be expressed as the angle between the two vectors.

[0262] Based on this, thresholds for length and angle differences can be set. If a certain difference (angle difference or distance difference) of a matching feature point exceeds the corresponding threshold, the point is considered to be an outlier or an inconsistent match and is removed from the final matching set.

[0263] Alternatively, it can be set to remove a point from the final match set only if the differences between two elements (angle difference and distance difference) of a matching feature point exceed the corresponding threshold.

[0264] In this embodiment, the grid partitioning verification method effectively removes inconsistent matching feature points caused by noise, occlusion, or non-rigid motion, thereby improving the quality of the final matching feature point set. This method is particularly suitable for scenarios with complex motion or uneven lighting changes. By localizing the image space, it can better adapt to local motion patterns, improving overall matching accuracy and robustness.

[0265] It should be noted that either verification operation one or verification operation two can be performed, or both can be performed. If both are performed, verification operation one or verification operation two can be performed first. Generally, verification operation one is performed first. After feature point tracking and preliminary RANSAC verification, in order to further improve the reliability and consistency of matching feature points, verification operation two is then performed, using a grid partitioning verification method for more detailed distance and angle verification.

[0266] Of course, in addition to the verification operations listed above, other verification operations can be used to further improve the reliability and consistency of matched feature points, such as feature matching verification, geometric constraint-based verification, and reverse optical flow checking. Feature matching verification verifies the reliability of the match by using methods such as the distance ratio of feature descriptors; geometric constraint-based verification verifies whether corresponding points satisfy geometric relationships by using fundamental or essential matrices. Besides RANSAC, there are other geometric constraint-based verification methods, such as the eight-point algorithm, five-point algorithm, etc.; reverse optical flow checking uses the position of the feature point found in the next frame to apply optical flow back to the previous frame and checks whether it returns to the origin; furthermore, neighborhood consistency or feature descriptors (such as SIFT descriptors) can be used to verify the reliability of the match. This paper does not specifically limit this method, and any method of verifying the reliability and consistency of matched feature points is applicable to the embodiments of this application.

[0267] The above is an introduction to the feature point extraction and feature point tracking process. For a summary, please refer to [link / reference]. Figure 9 As shown, it is a logical block diagram of a feature matching module in an embodiment of this application.

[0268] In the feature matching module, each video frame in the video to be processed is processed frame by frame. For the current frame (i.e. the video frame to be processed), if there is no previous video frame, or if the number of feature points in the previous video frame is not greater than the first number threshold, then feature point extraction and feature point tracking are performed on the video frame.

[0269] If the number of feature points in the previous video frame is greater than the first threshold, then feature point tracking is performed directly on that video frame.

[0270] Feature point extraction specifically includes processes such as acceleration segment test feature extraction for instruction set acceleration, mesh strategy, and line feature filtering; feature point tracking specifically includes processes such as optical flow tracking, RANSAC verification, and mesh partition verification, as detailed in the above embodiments, and will not be repeated here.

[0271] Specifically, based on the robust and fast front-end feature matching module proposed in this application, for each video frame in the video, please refer to... Figure 9 The logic shown processes each frame individually, and outputs more robust and stable inter-frame feature point association information and the position information of each feature point in consecutive frames through a variety of efficient strategies. Based on adaptive feature point detection, optical flow tracking and multi-strategy verification scheme, it provides stable and effective inter-frame feature point tracking pairs for subsequent spatial reconstruction.

[0272] The above methods not only improve the accuracy and robustness of feature point matching but also ensure the real-time performance and efficiency of the entire system. This is particularly important for real-time vision applications, enabling real-time and accurate visual processing tasks even with limited hardware resources, such as mobile phones. The technical solution in this application is especially suitable for mobile AR applications, significantly enhancing the object experience, such as more stable target tracking and faster scene understanding.

[0273] In this embodiment, the feature point extraction and feature point tracking processes described above can be implemented using a matching thread. Simultaneously, subsequent reconstruction operations can be performed in parallel using a reconstruction thread, allowing the matching thread and reconstruction thread to process the video in parallel, thereby improving the video processing speed.

[0274] Specifically, in this embodiment of the application, after feature point matching is performed on some frames in the video, the reconstruction operation can be started based on the detected video frames, while continuing to detect subsequent undetected video frames frame by frame.

[0275] When using a matching thread to perform frame-by-frame detection on a video, processing can be performed at a certain frame rate, such as 30 frames per second (30fps). For example, a 30-second video involves 900 frames. Feature point matching can be performed frame-by-frame on the first N frames (e.g., N=30, i.e., the first 30 frames) of the video. Then, a reconstruction thread is used to perform reconstruction operations based on the matching results of the first N frames. This includes initial reconstruction operations and incremental reconstruction operations performed sequentially. Specifically, the reconstruction thread first performs the initial reconstruction operation. After the initial reconstruction operation is completed, incremental reconstruction operations are performed iteratively. Each round of incremental reconstruction operations begins after the previous round of incremental reconstruction operations is completed.

[0276] During the reconstruction operation using the reconstruction thread, the matching thread continues to perform feature point detection on each remaining video frame. At the same time, the reconstruction thread also obtains the latest processing results from the matching thread in a timely manner to continue the reconstruction operation, gradually improving the reconstruction results, increasing efficiency and reducing memory consumption.

[0277] In this embodiment of the application, the specific execution process of S42 is as follows:

[0278] S42: During frame-by-frame detection, perform the following initial reconstruction operation S421 and incremental reconstruction operation S422 based on the detected video frames:

[0279] S421: The initial reconstruction operation is performed in the following manner: Based on the number of matching feature points and the inter-frame displacement, the first keyframe and the second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame.

[0280] In this context, each initial video frame within the range of the first keyframe to the second keyframe includes both the first and second keyframes. For example, if the first keyframe is frame 1 in the video and the second keyframe is frame 10, then each initial video frame within the range of the first and second keyframes is frames 1 to 10. Similarly, if the first keyframe is frame 2 in the video and the second keyframe is frame 20, then each initial video frame within the range of the first and second keyframes is frames 2 to 20; and so on.

[0281] In the embodiments of this application, the initial reconstruction operation processes the input video sequentially. This operation is implemented based on a new thread other than the matching thread, such as the reconstruction thread mentioned above, and various strategies are used to ensure the reconstruction success rate.

[0282] In practical applications, suitable frame pairs are first selected from the video sequence for initialization, and then the 3D structure of the scene is gradually constructed. One possible implementation method is to... Figure 10 The flowchart shown selects the first and second keyframes for initialization, including the following steps S101 to S108:

[0283] S101: Take the first frame of the video as the initial i-th frame and the second frame of the video as the initial j-th frame.

[0284] In this step, let i = 1 and j = 2. Here, "initial" means that i and / or j may change subsequently, as detailed below:

[0285] S102: For the current i-th frame and j-th frame, determine whether the number of each matching feature point and the inter-frame displacement meet the selection conditions.

[0286] Specifically, if the selection criteria are not met, then S103 is executed; if the selection criteria are met, then S104 is executed.

[0287] The selection criteria are: the number of matching feature points is greater than the third quantity threshold, and the average optical flow displacement of the matching feature points is greater than the preset displacement threshold.

[0288] The third quantity threshold is another pre-set threshold related to the number of feature points. It can be flexibly set according to factors such as experience, actual needs, computing resources, video content itself, and the accuracy requirements of 3D reconstruction. Similarly, the preset displacement threshold can also be flexibly set based on these factors. This article does not make specific limitations on this.

[0289] Specifically, if the number of matched feature points is greater than the third threshold, it means that there are enough matched feature points between the current i-th frame and the j-th frame. If the average optical flow displacement of the matched feature points is greater than the preset displacement threshold, it means that there is enough disparity between the current i-th frame and the j-th frame.

[0290] Therefore, based on this selection condition, a pair of frames (i, j) with sufficient disparity can be selected from the video sequence as the initial key frame pair, ensuring that the selected frame pair has enough matching feature points and significant motion changes, which is beneficial for subsequent pose estimation.

[0291] S103: Let j = j + 1, and return to S102.

[0292] Specifically, if the selection conditions are not met between the current i-th frame and the j-th frame, it means that (i, j) either does not have enough matching feature points or does not have significant motion changes. Therefore, this step is performed: slide j, let j = j + 1, and continue to try subsequent frames until a frame pair that meets the conditions is found, or until the current j-th frame is the latest / last frame in the video.

[0293] Conversely, if the above selection conditions are met between the current i-th frame and j-th frame, it means that there are enough matching feature points at (i, j) and there is a significant motion change, so step S104 is executed:

[0294] S104: Perform pose estimation on the current i-th frame and j-th frame, and determine the matching inliers between frames based on the pose estimation results.

[0295] In this embodiment of the application, two frames (i, j) with sufficient parallax are selected from the first frame of the video for monocular initialization. Therefore, starting from the first frame, the reconstruction thread will continuously monitor the number of matching feature points and the average optical flow displacement of each frame.

[0296] Based on this, if the number of matching feature points and the average optical flow displacement satisfy the above selection conditions, that is, if there are enough matching feature points in (i, j) and there is a significant motion change, then the current frame pair (i, j) is selected for pose estimation.

[0297] In summary, the process involves starting with frame i from the first frame and frame j from the second frame, counting the number of matching feature points from frame j to frame i, and the average optical flow displacement of all matching feature points. If the number of matching feature points is small or the average optical flow displacement is insufficient, frame j continues to the next frame; otherwise, initial pose estimation is performed on the current frames i and j.

[0298] Given the feature point matching in frames i and j, pose estimation can be performed directly based on the camera intrinsic parameters and the pose information of these matching feature points to recover the relative pose of frames i and j.

[0299] Specifically, pose estimation refers to estimating the camera's rotation and translation, i.e., the camera's position and orientation in 3D space, from a set of matched image feature points. It's important to note that if the input video does not provide camera intrinsics, estimation is required to initialize the camera intrinsics (fx, fy, cx, cy).

[0300] Taking a mobile electronic device like a smartphone as an example, the optical center (cx, cy) can be set to (weight / 2, height / 2) of the video resolution, i.e., the image center. The focal length (fx, fy) can be calculated based on empirical values ​​and image resolution, for example, set to the empirical value of a mobile phone camera (450*max(weight, height) / 640). This setting helps to initialize the camera without prior knowledge and obtain the initial camera intrinsic parameters.

[0301] Then, pose recovery is performed using the `findFundamentalMat` and `recoverPose` functions from the Open Source Computer Vision Library (OpenCV). In simple terms:

[0302] First, the OpenCV `findFundamentalMat` function is used to find the fundamental matrix (or essential matrix) between two frames. This matrix describes the relationship between matching feature point pairs between the two frames. Then, the `recoverPose` function is used to combine the fundamental matrix (or essential matrix) and the matching feature point pairs between the two frames to recover the relative pose between the two frames.

[0303] However, in the process of attempting to recover the relative pose between two frames, the matching feature point pairs used are not all correct; there may be incorrect matches. These incorrect matches are usually caused by factors such as image noise, lighting changes, dynamic objects, and repetitive textures. Therefore, after estimating the relative pose between two frames, it is necessary to check how many matching feature points support this pose to evaluate the pose recovery quality. Based on this, it can be determined whether to continue sliding i or j according to the pose recovery quality. Then, the most recently determined i-th frame is used as the first keyframe, and the most recently determined j-th frame is used as the second keyframe.

[0304] In this embodiment, the task can be accomplished by calculating the positions of all matching feature points reprojected onto another frame according to the estimated pose, and checking whether the distance between these projected points and the actual observation points is less than a certain predefined threshold. The number of matching feature points that is less than a certain predefined threshold (i.e., the supporting pose) is the number of matching inliers.

[0305] In practical applications, the matched interior points described above can be directly returned by recoverPose. Therefore, in this application, the pose calculation quality can be judged based on the number of matched interior points returned by recoverPose.

[0306] Specifically, the `recoverPose` function is used to recover the camera's relative pose information (rotation matrix and translation vector) from the fundamental matrix (or essential matrix) and matching feature point pairs. The function returns a boolean value indicating whether the recovery was successful, and a mask array indicating which matching feature point pairs were used to calculate the final pose. The non-zero elements in this mask array correspond to the matching inliers, i.e., those matching feature point pairs that satisfy geometric constraints (such as epipolar constraints).

[0307] The non-zero elements in the mask array refer to the following: if the distance between the projected point and the actual observation point is less than the predefined relevant threshold, then the point pair is considered to be consistent (matching interior points), and the corresponding position in the mask array is marked as a non-zero value.

[0308] In this embodiment of the application, the more matching interior points there are, the more point pairs support the calculated pose, which can be used as an indicator to evaluate the quality of pose calculation.

[0309] Therefore, if the number of recovered matching points is too small, it indicates that pose recovery has failed. In this case, different strategies will be adopted depending on the reason for the failure:

[0310] Based on the above idea, in an optional implementation, after S104, the following steps S105 to S108 are performed:

[0311] S105: Determine whether the total number of the determined matching intrapoints and the total number of the matching feature points have reached the corresponding thresholds.

[0312] In this step, based on the judgment result, it is specifically divided into the following cases: S106 (Case 1), S107 (Case 2), and S108 (Case 3):

[0313] S106: If the total number of matched intrapoints is greater than the fourth quantity threshold, then the current i-th frame is taken as the first keyframe and the current j-th frame is taken as the second keyframe.

[0314] For example, let the total number of matching feature points be n. all Matching interior points n in If n in Greater than the set fourth quantity threshold t in If the pose recovery is successful, the current frame pair is selected, with the current i-th frame as the first keyframe and the current j-th frame as the second keyframe.

[0315] Using the above method, suitable first and second keyframes can be selected, and the pose can be successfully recovered to obtain the rotation matrix describing the camera's orientation in 3D space and the translation vector representing the camera's position in 3D space. This information can be used to construct the camera's pose map and track its motion in the environment.

[0316] Specifically, what we obtain here is the relative pose of frames i and j, where the rotation matrix R i,j The transformation described is a 3×3 orthogonal matrix representing the rotation from the coordinate system of frame i to the coordinate system of frame j; the translation vector t i,j This describes the translation transformation from the coordinate system of frame i to the coordinate system of frame j, and is a 3D vector. In summary, this relative pose refers to the rotation and translation between the two frames relative to a common reference system (such as the world coordinate system or the reference coordinate system).

[0317] Since monocular vision cannot directly provide absolute scale (i.e. depth information), it is necessary to normalize the scale of the translation part, normalize the length of the translation vector to a unit length of 1, that is, keep the direction unchanged, but ignore the actual distance scale, so as to ensure the stability of subsequent numerical calculations.

[0318] Specifically, this process can involve adjusting the relative translation vector t between frames i and j. i,j Scale normalization can also be performed by selecting one of the two frames i and j as the reference coordinate system, setting its pose as the origin and unit rotation, and then determining the pose information of the other frame based on this, and then normalizing the translation vector of the other frame.

[0319] For example, if the current translation vector is t = [2, 3, 5] and its length is denoted as ||t||, then the normalized translation vector is t = [2 / ||t||, 3 / ||t||, 5 / ||t||].

[0320] If n in Not greater than the set fourth quantity threshold t in If the current pose recovery fails, then it is considered that the current pose recovery has failed.

[0321] In this embodiment, after pose recovery fails, there are two adjustment methods to try new frame pairs, specifically the following two cases:

[0322] S107: If the total number of matched internal points is not greater than the fourth quantity threshold and the total number of matched feature points is not greater than the fifth quantity threshold, then let i = i + 1 and return to S102.

[0323] Specifically, if n in Not greater than the set fourth quantity threshold t in And n allNot greater than the set fifth quantity threshold t all If the number of matching feature points is insufficient, it may be due to insufficient environmental texture or poor image quality. In this case, slide i and return to S102 to reselect two frames for initialization.

[0324] S108: If the total number of matched internal points is not greater than the fourth quantity threshold, and the total number of matched feature points is greater than the fifth quantity threshold, then increase the preset displacement threshold, let j = j + 1, and return to S102.

[0325] Specifically, if n in Less than the set fourth quantity threshold t in , but n all Greater than the set fifth quantity threshold t all If the scene is deep enough, it is considered a scene with a large depth of field. Because the scene depth is large, the parallax between two frames is very small, making it difficult to accurately estimate the pose. Therefore, the preset displacement threshold T is increased. auto Slide j and return to S102 to continue, reselecting frame pairs with greater disparity to improve the accuracy of pose estimation.

[0326] The fourth and fifth quantity thresholds are also pre-set thresholds related to the number of feature points. They can be flexibly set based on factors such as experience, actual needs, computing resources, the video content itself, and the accuracy requirements of 3D reconstruction. This paper does not impose specific limitations on them. Furthermore, because the fourth quantity threshold is related to matching interior points, and the fifth quantity threshold is related to matching feature points, the fourth quantity threshold is generally smaller than the fifth quantity threshold.

[0327] In this application embodiment, there are many ways to increase the preset displacement threshold. Several are briefly listed below:

[0328] Method 1: Fixed Increment Method

[0329] Specifically, each time it can be done at T auto Based on this, add a fixed value, for example, 5 pixels each time, then T auto =T auto +5.

[0330] Method 2: Proportional Adjustment Method

[0331] Specifically, it can also be increased according to the current preset displacement threshold ratio, for example, by 20% each time, then T auto = (1 + 20%)T auto .

[0332] The two methods listed above are just simple examples. Any method of increasing the preset displacement threshold is applicable to the embodiments of this application, and will not be described in detail here.

[0333] In the above implementation, reconstruction begins when the number of processed feature points and the length of the optical flow trajectory exceed the threshold. An adaptive keyframe filtering strategy is adopted, and by adding temporal constraints, the pose and spatial 3D points of each frame are quickly restored.

[0334] In summary, the above details the selection of the first and second keyframes. Based on this, 3D reconstruction can be performed on each initial video frame within the range of the first to the second keyframes. This 3D reconstruction process requires the pose information corresponding to these frames, as well as the association and pose information of feature points between two or more frames, that is, the matching feature points between frames and their position information in the corresponding video frames obtained in S41.

[0335] The following section explains the calculation of pose information and the 3D reconstruction process for these frames:

[0336] An optional implementation involves determining the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe in the following manner:

[0337] First, based on the camera intrinsic parameters and the position information of each matched feature point between the first and second keyframes, the relative pose between the first and second keyframes is determined. Based on the relative pose, the pose information corresponding to the first and second keyframes is determined with the first keyframe as the reference coordinate system. On this basis, the pose information corresponding to other initial video frames between the first and second keyframes can be determined frame by frame, based on the pose information corresponding to the first and second keyframes, combined with the uniform motion model and the projection constraints between 3D and 2D points.

[0338] When calculating the pose information of the first and second keyframes, since the input video does not provide camera intrinsic parameters, it is necessary to estimate the initial camera intrinsic parameters. Then, the pose recovery can be performed using OpenCV's findFundamentalMat and recoverPose functions, as detailed in the above embodiment.

[0339] The pose recovery obtained here is the relative pose of frames i and j, including the rotation matrix R. i,j Translation vector t i,j Then, the i-th frame can be selected as the reference coordinate system (such as the world coordinate system), and its pose information can be set to the origin and unit rotation, i.e., R. i = I (identity matrix), t i = [0, 0, 0].

[0340] Based on this, the pose information of the j-th frame relative to the reference coordinate system can be calculated, i.e., R. j =R i,j Ri , t j =R i,j t i +t i,j Because R is set i =I and t i = [0, 0, 0], therefore R j =R i,j and t j =t i,j .

[0341] In addition, considering that monocular vision cannot directly provide absolute scale (i.e., depth information), it is necessary to normalize the scale of the displacement part. For details, please refer to the above embodiments. Repeated parts will not be repeated.

[0342] For the remaining frames between i and j, the pose can be solved frame by frame based on the 3D-2D PnP optimization method (this method is denoted as module1).

[0343] Specifically, the main idea of ​​this solution process is based on the assumption of uniform motion and 3D-2D projection constraints. Before optimization, the assumption of uniform motion is used to estimate the initial pose of the current frame. Then, the known 3D point cloud and the 2D feature points of the current frame are matched, and the pose information corresponding to the current frame is optimized by minimizing the projection error.

[0344] The uniform motion model is a model used to estimate the pose change of a camera between different time points. Its main idea is the uniform motion assumption, which assumes that the camera makes uniform linear motion or uniform rotation within a short time interval. In the embodiments of this application, the uniform motion model is adopted, and the motion speed of the current frame can be estimated using the pose information of the previous frame. The estimated pose information of the current frame is used to optimize the initial value and improve the stability of the solution.

[0345] Specifically, given the pose T of two consecutive frames l-2 and T l-1 It can be assumed that during the time intervals (time1) and (time2) between frame l-2 and frame l-1, and between frame l-1 and the current frame (frame l), the camera performs uniform linear motion and uniform rotation.

[0346] Based on this assumption, the pose information T of the current frame l can be estimated. l Including translation vector t l and rotation matrix R l The translation vector t of the current frame l and rotation matrix R l The calculation formula is as follows:

[0347]

[0348] R l =R l-1 (Formula 3)

[0349] Among them, t l-2 , t l-1 R represents the translation vectors of frames l-2 and l-1 in the world coordinate system. l R l-1 Let `time1` and `time2` represent the rotation matrices of frames `l-2` and `l-1` in the world coordinate system, respectively, and `time1` and `time2` represent the times from frame `l-2` to frame `l-1` and from frame `l-1` to frame `l`, respectively. Equations 2 and 3 indicate that the translation component of the current frame is predicted by calculating the average velocity between the previous two frames, while the rotation component remains unchanged. This step helps improve the convergence speed and stability of the optimization.

[0350] In this process, 3D-2D projection constraints can be used to establish a loss function, and an optimization method with a loss function can be used to solve the problem, so as to reduce the error caused by mismatch and improve robustness.

[0351] Specifically, the optimization target is the pose information T corresponding to the current frame. l The constraint term is the error of the 2D projection of the 3D point to the current frame. The projection equation (also known as the observation equation) is Equation 4 below. Specifically, by matching the feature points in the current frame with their corresponding 3D world coordinates, these 3D coordinates are projected onto the current frame image to obtain 2D projected coordinates, which are then compared with the actual observed 2D feature point coordinates.

[0352] The loss function can be the Cauchy loss function. Cauchy loss is suitable for processing datasets containing outliers. Its characteristic is that the growth rate of the loss function gradually slows down as the residual increases, thereby reducing the impact of outliers and making the optimization more robust. Based on this, this application can use nonlinear optimization methods (such as the Levenberg-Marquardt algorithm) to minimize the loss function, thereby adjusting the pose information T corresponding to the current frame. l This allows it to better conform to 3D-2D projection constraints, and the final output is the optimized current frame pose information T. l It is more accurate than the results calculated directly based on the assumption of uniform motion.

[0353] Of course, in addition to Cauchy Loss, other loss functions can also be used, such as Huber Loss, Geman-McClure Loss, Tukey Loss, etc.

[0354] Huber Loss is a loss function that combines the advantages of Mean Squared Error (MSE) and Mean Absolute Error (MAE). It exhibits quadratic characteristics when the error is small, and linear characteristics when the error is large. This retains sensitivity to small errors while reducing the impact of large errors, making it suitable for situations where the data may contain some outliers but generally follows a Gaussian distribution.

[0355] Geman-McClure Loss is a non-linear loss function that is highly robust to outliers. Its shape is similar to Huber Loss, but it performs more smoothly in extreme cases and performs well with noisy data, especially at high noise levels.

[0356] Tukey Loss is very insensitive to outliers and is suitable for processing datasets with a large number of outliers. It can effectively reduce the impact of outliers on the optimization process.

[0357] It should be noted that the choice of loss function listed above depends on the specific application scenario and the characteristics of the dataset. For example, if a large number of outliers are expected (e.g., feature matching errors due to rapid movement), Tukey Loss can be chosen to enhance robustness; if the goal is to reduce the impact of outliers while maintaining accuracy, Huber Loss can be considered, and so on. The loss functions listed above are just simple examples, and this article does not impose specific limitations on them.

[0358] Based on the above implementation method, the pose information corresponding to other initial video frames between the first and second keyframes can be determined frame by frame. The uniform motion assumption simplifies the pose estimation process, making the computation more efficient, and ensures that pose changes between adjacent frames are smooth. This is helpful for subsequent optimization steps. Furthermore, by using 3D-2D projection constraints and a suitable loss function (such as Cauchy Loss), the influence of outliers can be effectively reduced, improving the robustness of pose estimation. The entire process combines the simplicity of the uniform motion assumption with the accuracy of nonlinear optimization based on 3D-2D projection, effectively estimating the camera's pose changes between adjacent frames.

[0359] Through the above process, the pose information corresponding to all frames from i to j can be obtained. However, in the initial reconstruction operation, the purpose is to perform 3D reconstruction. Triangulation of feature points can convert 2D observation points from different perspectives into points in 3D space, thereby constructing a 3D model of the scene. Therefore, in the process of calculating pose information, it is also necessary to combine triangulation processing to recover several 3D points, thereby forming an initial map (a 3D point cloud map). An optional implementation method is as follows:

[0360] First, after obtaining the relative poses of the first keyframe and the second keyframe, the first keyframe can be used as a reference coordinate system to determine the pose information corresponding to the first keyframe and the second keyframe. Then, based on the pose information corresponding to the first keyframe and the second keyframe, and combined with the position information of each matching feature point between the first keyframe and the second keyframe, the matching feature points between the first keyframe and the second keyframe are subjected to two-frame triangulation.

[0361] In the initial reconstruction operation, the first two frames, i and j (i.e., the first keyframe and the second keyframe), are selected and their poses are restored. Then, the initial reconstruction is performed on the two frames i and j whose relative poses have been restored. Since the pose information of the two frames is known, the initial map can be directly restored by triangulation. Here, the two-frame triangulation is directly used (this method is denoted as module2).

[0362] Two-frame triangulation refers to recovering the coordinates of corresponding feature points (i.e., matched feature points) in three-dimensional space from two frames of images. Specifically, the world coordinates (x, y, z) of the corresponding 3D point can be calculated using the relative pose and the pixel coordinates (u, v) of the feature points in the two frames, that is, the coordinates in the world coordinate system, so as to build a global map.

[0363] In the embodiments of this application, the world coordinate system can be an arbitrarily selected global coordinate system, or it can be the camera coordinate system of one frame as a temporary reference, and then transformed to the global coordinate system through pose transformation. For example, in this paper, the i-th frame is used as a reference.

[0364] In summary, firstly, given the 2D observations of feature points and their corresponding camera intrinsics and poses, the homogeneous coordinates of the feature points are calculated using a projection matrix. Then, the homogeneous coordinates of multiple observed 2D points are combined to form a system of linear equations, which are then solved to obtain the homogeneous coordinates of the 3D points. Finally, the recovered 3D points are deleted, removing points with depth anomalies. In computer vision, each camera observation typically refers to the pixel coordinates of a feature point on the image. For a feature point, its (u) and (v) coordinates can be measured on the image; this is what is known as 2D observation.

[0365] The specific method is as follows: set the homogeneous coordinates of the feature points to be calculated. The homogeneous coordinates of the observation point in the 2D image of the l-th camera (i.e., the l-th video frame image in this paper) are: The projection matrix is ​​P l =K[R l |t l ], Therefore, the projection model is as follows:

[0366]

[0367] Among them, R l and t l For the l-th camera (i.e., the l-th video frame), K represents the orientation (i.e., the rotation matrix, which is a 3×4 matrix) and position (i.e., the translation vector, which is a 3×1 matrix). K is the camera intrinsic parameter, and K[R|t] is defined as the projection matrix P, which is the product of the camera intrinsic parameter and the camera pose (rotation matrix and translation vector). These represent the first, second, and third rows of the projection matrix, respectively. Projection matrix P l Indicates 3D points 2D image coordinates projected onto the l-th camera Above. In this paper, the last component "1" in the two homogeneous coordinates is the scaling factor (w). The actual 3D coordinates are obtained by normalizing the homogeneous coordinates (letting w = 1). Letting w = 1 is to be able to represent the point at infinity and avoid the division by zero problem.

[0368] Therefore, the projection matrix P l Multiply by the homogeneous coordinates of the feature point This will generate a 3×1 vector, the first two components of which should be the pixel coordinates (u) of the feature point in the l-th video frame. l v l The third component is 1. Therefore, for this feature point... Each camera observation can provide two constraints (corresponding to two coordinates u and v), which can be transformed into linear equations, so each camera observation has two linear equations.

[0369] Specifically, based on Formula 4 above, let Cross-multiply both sides of the equation. Vector, then (The cross product of two identical vectors results in a zero vector). Based on this, the following two linear equations can be derived:

[0370]

[0371] Similarly, by combining the constraint equations of M camera observations (representing the number of observations at that point) into 2M linear equations, that is, by letting the value of l in the above formula 5 be from 1 to M, we can obtain the following formula 6:

[0372]

[0373] In the embodiments of this application, triangulation is essentially solving the system of linear equations in Formula 6 above. This system of equations is composed of the projection equations of each camera. For the case of two-frame triangulation, M=2, there are only the first 4 linear equations. For the case of multi-frame triangulation, M>2, there will be more than 4 linear equations.

[0374] By using mathematical tools such as the least squares method to solve the homogeneous system of equations in Formula 6 above, we can obtain the following solution: This yields the 3D coordinates of the point; the above describes the feature point triangulation process. This operation is performed on each feature point that needs to be recovered (referring to the matching feature points between two frames). Additionally, anomaly detection can be performed using the following methods:

[0375] Anomaly Validation 1: Perform depth value validation on each recovered 3D point, i.e., determine the depth of the recovered 3D point under the pose of each observation camera. This depth value refers to the distance from the 3D point to the camera, which is obtained from the depth map or during the triangulation process. When performing depth value validation, points with negative depth values ​​and / or excessively far points (points with depths greater than a threshold, which can be simply referred to as far points) need to be removed.

[0376] Negative depth values ​​typically indicate calculation errors, likely caused by errors in calculation or sensor noise. Far points may be unreliable due to decreased accuracy or unstable calculations. Such points are outside the reasonable range and are also considered untrustworthy. Therefore, negative values ​​and far points are considered abnormal and invalid points. By removing such points, the validity and accuracy of 3D points can be ensured, thereby improving the reliability and robustness of the map.

[0377] During this process, triangulation of two frames can yield multiple 3D points, which are points observed jointly by the i-th and j-th frames. These 3D points form a 3D point cloud shared by the i-th and j-th frames. This point cloud describes the true location of objects or scene elements in the scene that are jointly observed by the i-th and j-th frames.

[0378] As more frames are added and more triangulation operations are performed, such as subsequent two-frame and multi-frame triangulation of other frames, the 3D point cloud gradually becomes denser and more complete, reflecting more details in the scene. In short, these 3D points do not belong to a specific frame, but to the entire scene; they are identified and located from the shared perspective of two or more frames. The subsequent two-frame and multi-frame triangulation processes for other frames follow the same principle, which will not be elaborated upon here.

[0379] The following is a brief description of the subsequent two-frame triangulation and multi-frame triangulation processes:

[0380] Subsequently, for other initial video frames between the first and second keyframes, 3D-2D optimization can be performed based on a uniform motion model to determine the corresponding pose information frame by frame. After determining the pose information corresponding to each other initial video frame, based on the pose information corresponding to this other initial video frame and its previous video frame, and combined with the position information of each matching feature point between this other initial video frame and its previous video frame, two-frame triangulation processing is performed on each matching feature point between this other initial video frame and its previous video frame.

[0381] In the 3D-2D PnP (module 1) optimization based on the uniform motion model, the optimization method with loss function is used to solve the problem, so as to reduce the error caused by mismatch and improve robustness. The specific calculation method is as above, and will not be repeated here.

[0382] In this process, the pose information of other initial video frames between frames i and j is obtained, and then the other initial video frames and their previous frames are triangulated into two frames (module2). The specific calculation process is the same as above, and will not be repeated here.

[0383] During this process, triangulation of two frames can yield multiple 3D points, which are points commonly observed between the other initial video frame and the previous frame. These 3D points form a common 3D point cloud between the two frames, which describes the true location of objects or scene elements in the scene that are commonly observed by the two frames.

[0384] Once all frames between frames i and j have been calculated, we can obtain the pose information corresponding to all frames between i and j, as well as several 3D points, which form the initial map and serve as the initial 3D reconstruction result.

[0385] Since the above process involves triangulation between two frames, it is prone to failure, resulting in a poor initial map accuracy. In this embodiment, at least one of the following methods can be used to optimize the initial map:

[0386] Optionally, perform at least one of the following optimization operations on the initial 3D reconstruction results:

[0387] Optimization Operation 1: Based on the pose information corresponding to each initial video frame within the range from the first keyframe (frame i) to the second keyframe (frame j), and combined with the position information of each matching feature point between each initial video frame, perform multi-frame triangulation processing on each matching feature point between each initial video frame; add the 3D point cloud obtained through multi-frame triangulation processing to the initial 3D reconstruction result to optimize the initial 3D reconstruction result.

[0388] This optimization operation improves the accuracy of the initial map by performing multi-frame triangulation (module 3) to supplement feature points.

[0389] Similar to the two-frame triangulation described above, multi-frame triangulation refers to recovering the coordinates of corresponding feature points (i.e., matching feature points) in three-dimensional space from multiple frames of images. Specifically, the world coordinates (x, y, z) of the corresponding 3D point for each matching feature point pair can be calculated using the relative pose and the pixel coordinates (u, v) of the feature points in multiple frames, that is, the coordinates in the world coordinate system, in order to optimize the global map.

[0390] In this embodiment, the basic principle of multi-frame triangulation is also Equation 6. For each feature point, M frames are observed, and the common 2D points of the M observation frames are collected and projected to form 2M linear equations. The coordinates of the 3D points are then solved. During the initial reconstruction operation, the value of M represents the sum of the initial video frames within the range from the first keyframe (frame i) to the second keyframe (frame j). For example, if i = 1 and j = 10, then M = 10. Based on Equation 6, 20 linear equations can be constructed to perform multi-frame triangulation on the common matching feature points between frames 1 to 10. Alternatively, if i = 2 and j = 20, then M = 19. Based on Equation 6, 38 linear equations can be constructed to perform multi-frame triangulation on the common matching feature points between frames 2 to 20; and so on.

[0391] Based on the above idea, the solution to the homogeneous equation system is obtained. This yields the 3D coordinates of the point; the above describes the feature point triangulation process. This operation is performed on each feature point that needs to be recovered (referring to matching feature points common across multiple frames). Additionally, anomaly detection can be performed using at least one of the following methods:

[0392] Anomaly Validation 2: Perform depth value validation on each recovered 3D point, and remove negative values ​​and / or far points, similar to Anomaly Validation 1 above. For details, please refer to the above embodiment, and it will not be repeated here.

[0393] Anomaly check 3: Perform stability elimination, that is, check whether the matrix condition number is greater than the threshold. If it is greater than the threshold, return failure; otherwise, succeed.

[0394] Specifically, after multi-frame triangulation, the stability of 3D point reconstruction can be evaluated by checking if the condition number of the matrix is ​​greater than a threshold. A larger condition number indicates a less stable matrix, potentially amplifying 3D point coordinate errors. If the condition number is greater than the threshold, 3D point reconstruction is considered a failure; if it is less than or equal to the threshold, 3D point reconstruction is considered successful and stable. This verification helps ensure the accuracy of 3D point reconstruction.

[0395] It should be noted that the thresholds mentioned in the above anomaly detection methods can be flexibly set based on experience, and the specific setting method is similar to that of other thresholds. This article does not impose specific limitations. In addition, the other thresholds in this article are similar, and will not be elaborated on one by one.

[0396] To increase the number of 3D points in the initial map, a multi-frame triangulation method is used. During the solution process, depth deletion is also performed to remove unreasonable 3D points, thereby improving the success rate and accuracy of triangulation.

[0397] After the optimization operation is completed, more 3D points can be obtained in the initial map to improve the accuracy of the initial map.

[0398] Optimization Operation 2: Perform global BA optimization on the current initial 3D reconstruction results.

[0399] This optimization operation refers to performing the first global BA, which optimizes the accuracy of the initial map. Global BA is a non-linear optimization method that aims to refine the estimation of camera pose and 3D point coordinates by minimizing projection errors, thereby reducing projection errors and improving reconstruction accuracy.

[0400] In this embodiment, the optimization terms for the initial global BA optimization are: pose information corresponding to all frames between the first keyframe (the current i-th frame) and the second keyframe (the current j-th frame), the positions of all current 3D points, and the camera intrinsic parameter K (previously K is set to an empirical value, such as the initial camera intrinsic parameter used above), adding pose continuity constraints. The constraint terms are: 2D point constraints from 3D points to all observation frames (here referring to all frames from i to j), and the projection equation is Equation 4. Specifically, by matching the feature points in the observation frames with their corresponding 3D world coordinates, these 3D coordinates are projected onto the observation frame image to obtain 2D projected coordinates, which are then compared with the actual observed 2D feature point coordinates.

[0401] Furthermore, it should be noted that due to the lack of scale information in monocular vision and the inability to observe 7 degrees of freedom, there is no reference point to determine the absolute position of the camera in the first frame to be optimized. Similarly, in the second frame to be optimized, the actual movement distance of the camera relative to the first frame cannot be determined, making it impossible to determine the actual scale of the translation portion corresponding to the first and second frames to be optimized. Therefore, this application needs to fix the translation portion of the first and second frames to be optimized during global BA optimization to ensure that the problem is observable.

[0402] Of course, in addition to the above method of not optimizing the pose of the first frame and the translation parameters between the second frame, a scale factor can be introduced during the optimization process. This factor is multiplied by the translation vector. In this way, even if the absolute distance of the translation is unknown, the scale factor and rotation parameters can still be optimized to obtain the optimal relative pose estimation, and so on.

[0403] Meanwhile, considering that there is usually a large overlap between adjacent frames in 3D reconstruction, this means that the camera movement between adjacent frames is usually relatively small. This small movement and rotation change can be regarded as continuous, that is, the camera will not suddenly jump to a completely different position or direction between adjacent frames. Therefore, in addition to Equation 4 above, this application further introduces pose continuity constraints, namely adjacent frame constraints, in the global BA optimization to ensure that pose changes between adjacent frames are smooth.

[0404] In this embodiment, the goal of global BA is to reduce accumulated errors and ensure that the pose and 3D point coordinates of all frames are as consistent and accurate as possible. Pose continuity constraints help prevent drastic fluctuations during the optimization process and ensure that the results are more stable and reasonable.

[0405] In summary, the loss function for global BA optimization is shown in Formula 7 below:

[0406] loss=(norm((t l+1 -t l )-(t l -t l-1 )),if norm((t l+1 -t l )-(t l -

[0407] t l-1 ))>Thres), else loss=0 (Formula 7)

[0408] In formula 7 above, t l-1 , t l , t l+1Let Thres represent the translation vectors of frames l-1, l, and l+1 in the world coordinate system. Thres is a flexibly set threshold related to the translation vector in this embodiment. In practical applications, the selection of Thres needs to be adjusted according to the specific application scenario and dataset characteristics to achieve the best balance, neither excessively suppressing reasonable pose changes nor tolerating excessive pose abrupt changes. The specific setting method is similar to that of other thresholds, and this paper does not make specific limitations.

[0409] This loss function states that if the difference between two adjacent translation changes exceeds a predefined threshold Thres, the function calculates the norm of this difference as the loss value. This indicates that a sudden increase in pose change between adjacent frames may not conform to the assumption of continuous camera motion. If the difference is less than or equal to the threshold Thres, the loss is 0, meaning the pose change is considered reasonable and without abrupt changes.

[0410] Among them, the pose changes between adjacent frames suddenly increase, which may be caused by noise, feature matching errors or algorithm instability.

[0411] This loss function penalizes solutions that cause abrupt pose changes during optimization, thereby controlling the continuity and stability of pose changes between adjacent frames and ensuring smooth camera motion. This mechanism helps filter out abnormal pose changes caused by noise or algorithmic instability, improving the reliability and robustness of the final initial reconstruction structure.

[0412] After global Business Analytics (BA) optimization is completed, at least one of the following anomaly checks can be performed:

[0413] Anomaly Detection 4: Feature Point Anomaly Removal. Specifically, this involves verifying the projection error of each 3D point on each observation frame after optimization. This error can be calculated using metrics such as Euclidean distance or Mahalanobis distance. Points with projection errors exceeding a threshold are removed (i.e., points with excessively large projection errors are removed) to ensure the accuracy and robustness of the map. This threshold is preset based on application requirements and image resolution; this paper does not specify a concrete limit for it.

[0414] In this embodiment, projection error refers to the difference between the theoretical projection position of a 3D point on the image plane based on the currently estimated camera pose and its actual observed position on the image. If the projection error of a 3D point exceeds a threshold in multiple frames, then this point may be inaccurate. The reasons may include feature matching errors, motion blur, occlusion, lighting changes, camera calibration errors, etc. Such points need to be deleted to avoid affecting the overall reconstruction accuracy.

[0415] Anomaly Validation 5: Depth Value Validation. Specifically, depth values ​​are validated for each optimized 3D point to remove negative values ​​and / or distant points, thereby improving the reliability and robustness of the map. Similar to Anomaly Validation 1 and Anomaly Validation 2 above, please refer to the above embodiments for details, which will not be repeated here.

[0416] For example, after global BA optimization is completed, anomaly removal of feature points and depth value verification can be performed simultaneously to improve the reliability and robustness of the map.

[0417] It should be noted that optimization operation one and optimization operation two can be performed selectively, or both can be performed. If both are performed, optimization operation one can be performed first, or optimization operation two can be performed first. Generally, optimization operation one is performed first. After supplementing feature points through multi-frame triangulation, global BA can be used for further optimization to improve the accuracy of the initial map.

[0418] In the above implementation, the entire process, from single-frame pose estimation to multi-frame triangulation and then to global BA optimization, progressively builds and refines the 3D map, ensuring the accuracy and completeness of the reconstruction. Furthermore, by introducing depth deletion, pose continuity constraints, and selecting appropriate loss functions, this process can effectively reduce errors and improve reconstruction quality when dealing with complex real-world scenes.

[0419] In this embodiment of the application, considering that the first keyframe is not the first frame in the video, the following operations can also be performed during the initial reconstruction operation:

[0420] For the other video frames before the first keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other video frames and the next video frame, and combined with the position information of each matching feature point between the other video frames and the next video frame, two-frame triangulation processing is performed on each matching feature point between the other video frames and the next video frame; and the three-dimensional point cloud obtained through the two-frame triangulation processing is added to the current initial three-dimensional reconstruction result to optimize the initial three-dimensional reconstruction result.

[0421] Specifically, after the initial reconstruction is completed in the manner described above, the pose is first restored for the frames preceding frame i, and the two frames are triangulated. For details, please refer to the relevant embodiments in module 1 and module 2, which will not be repeated here. During this process, if a frame fails to be solved, the solution for the preceding frame is abandoned. By this point, the pose of all frames before frame j has been solved, and several 3D points are obtained.

[0422] For example, if the first keyframe (i-frame) is the 5th frame in the video, then the other video frames before the first keyframe include the 1st, 2nd, 3rd, and 4th frames in the video. In this process, the pose information of the 4th frame can be calculated first. Based on this, two-frame triangulation is performed on the 4th and 5th frames. Then, the pose information of the 3rd frame is calculated, and two-frame triangulation is performed on the 3rd and 4th frames. Next, the pose information of the 2nd frame is calculated, and two-frame triangulation is performed on the 2nd and 3rd frames. Finally, the pose information of the 1st frame is calculated, and two-frame triangulation is performed on the 1st and 2nd frames.

[0423] In the above implementation, the initial 3D reconstruction results can be further optimized by performing pose recovery and two-frame triangulation on the frames preceding frame i. During this process, if a frame fails to be solved, the solution of the preceding frame is abandoned. This improves robustness, ensures reconstruction quality, saves computational resources, simplifies the processing flow, and ensures that the obtained 3D points are more reliable and accurate.

[0424] In this embodiment, after initial 3D reconstruction and preliminary 3D points are obtained based on frame j and its preceding frames, a new stage is entered: plane detection and refinement. This process aims to identify planar surfaces that may exist in the video scene, which is particularly important for AR applications because it allows virtual objects or information to be stably placed on the plane of the actual scene.

[0425] Specifically, in AR scenarios, plane detection is a crucial component because it helps identify stable surfaces within the scene, allowing virtual objects to be placed on these surfaces. For example, detecting planes such as floors and walls in a room allows for the placement of virtual navigation markers, enabling AR indoor navigation; detecting planes such as tables and floors allows for the placement of virtual game characters, enabling AR games; detecting planes in a room allows users to place virtual furniture, enabling AR furniture arrangement; and so on. Of course, other application scenarios are also applicable to the embodiments of this application, including but not limited to interior design scenarios, exterior architectural design scenarios, game scene design scenarios, and intelligent map navigation scenarios, which will not be elaborated upon further here.

[0426] Therefore, in the initial reconstruction operation of this application embodiment, after obtaining the three-dimensional point cloud (sparse point cloud, containing several 3D points), the initial plane detection step can be entered, and plane fitting can be performed using K-Dimensional tree (KD tree) and random sampling.

[0427] Specifically, the first step is to construct a KD-tree from all 3D points in the obtained 3D point cloud. A KD-tree is an efficient binary tree data structure used to organize and index high-dimensional data. Its basic idea is to divide the data space into hyperrectangular regions, with each node representing a hyperrectangular region. In this embodiment, the KD-tree is mainly used to accelerate the query and search of point cloud data, especially for subsequent nearest neighbor point queries.

[0428] In simple terms, the construction process of a KD tree is as follows: the current 3D point cloud data space is divided into a left half-space and a right half-space, with the partitioning axis alternating along the dimensions. Subtrees are recursively constructed in both the left and right half-spaces until all 3D points are contained in a single leaf node. The specific construction method is not specifically limited in this paper.

[0429] Based on the construction of the KD-tree, planar detection can be performed. One possible detection method is as follows:

[0430] The plane detection operation proposed in this application is executed iteratively until the preset number of iterations m is reached, or the plane cannot be detected in the latest round of plane detection operation.

[0431] Each round of plane detection operation executes the following steps a to f:

[0432] Step a: Randomly sample n 3D points from the current 3D point cloud that are not currently assigned to a plane.

[0433] The preset number of iterations m and the number of 3D points n selected in each iteration can be flexibly set based on factors such as experience, actual needs, computing resources, video content itself, and the accuracy requirements of 3D reconstruction. This paper does not impose specific limitations on these.

[0434] In addition, when n is large enough, it can be guaranteed that no plane will be detected in a certain round of plane detection operation. Therefore, the subsequent iteration process will most likely not detect any plane either, so the iteration of the plane detection operation can be terminated.

[0435] For example, if m = 10 and n = 100, it means that after obtaining the 3D point cloud through the above method during the initial reconstruction operation, it can be iterated to multiple rounds of plane detection operation. In each round of plane detection operation, 100 3D points that are not currently assigned to any plane can be randomly sampled from Sa as seed points for potential planes, and then step b is executed:

[0436] Step b: For each of the n randomly sampled 3D points, perform the following planar sampling operations, including sub-steps b1 to b4:

[0437] Sub-step b1: For a 3D point, a candidate plane is obtained by fitting a preset number of nearest neighbors of this 3D point in the current 3D point cloud.

[0438] Assuming the preset number is q, this means that for each 3D point selected in step a, its corresponding q nearest neighbor points (nearest neighbor points are also 3D points) are determined. Similar to m and n above, the value of q can be set flexibly, and this article does not impose specific limitations on it.

[0439] When searching for the nearest neighbors of a 3D point, since a KD tree has been constructed based on the current 3D point cloud, the nearest neighbor search can be performed through the KD tree in this sub-step. The KD tree can quickly find the nearest point to it, thus obtaining the q nearest neighbors of the 3D point. These q nearest neighbors form the point set p.

[0440] In this embodiment, the KD-tree can significantly reduce search time and accelerate nearest neighbor search, thereby improving the efficiency of plane detection. Furthermore, the KD-tree quickly finds q nearest neighbors for each seed point; these points are used to fit a preliminary plane, denoted as the candidate plane. The specific fitting process is as follows:

[0441] The equation of the spatial plane is Ax + By + Cz + D = 0, where A, B, C, and D are constants (to be solved) representing plane parameters, and A, B, and C are not simultaneously zero. Therefore, when fitting the plane to the point set p, we can assume C = -1, then Ax + By - z + D = 0, thus constructing the following formula 8:

[0442]

[0443] In Formula 8, (x, y, z) represent the 3D coordinates of q points in point set p. For example, (x1, y1, z1) represents the 3D coordinates of the q-th point in point set p, (x2, y2, z2) represents the 3D coordinates of the 2nd point in point set p, ..., (x... q y q , z q ) represents the 3D coordinates of the q-th point in the point set p.

[0444] Based on the above formula 8, the most suitable plane parameters A, B, C, and D can be found by using the least squares method to minimize the sum of the squared distances from all points in the point set p to this plane. The plane determined by these plane parameters is a candidate plane in the embodiments of this application.

[0445] Sub-step b2: Determine whether the number of points in the candidate plane is higher than the preset threshold for the number of points in the plane.

[0446] Among them, the points in the plane are the three-dimensional points that are not currently assigned to the plane and whose distance from the candidate plane is less than a preset distance threshold.

[0447] Note that this process involves identifying temporary in-plane points corresponding to a candidate plane from all 3D points in the current 3D point cloud that are not assigned to any plane, based on the distance from each point to the plane. For example, in-plane points corresponding to other target planes determined before this iteration no longer belong to the 3D points of the currently unassigned plane; and so on.

[0448] Similar to the other threshold parameters mentioned above, the preset distance threshold is a distance-related value, and the preset plane point threshold is a minimum plane point threshold set in this paper. These thresholds can also be set flexibly, and this paper does not impose specific limitations on them.

[0449] In this sub-step, for each 3D point in the current 3D point cloud that does not belong to a plane, the distance from the 3D point to the candidate plane is calculated. If the distance is less than a preset distance threshold, the 3D point is considered to belong to the candidate plane, and the 3D point can be determined to be a point in the plane corresponding to the candidate plane.

[0450] The formula for the distance from a 3D point to the candidate plane Ax+By+Cz+D=0 can be specifically shown below:

[0451]

[0452] Where (x0, y0, z0) are the spatial coordinates of the three-dimensional point, and A, B, C and D are the planar parameters of the candidate plane.

[0453] Next, analyze whether the number of points in the determined plane exceeds the set minimum threshold for the number of plane points. In this sub-step, if the result is yes, it indicates that enough support points have been found to confirm the existence of this candidate plane, and the candidate plane can continue to be optimized or recorded, specifically executing sub-step b3; otherwise, the fitted candidate plane does not have enough support points, and then sub-step b4 is executed:

[0454] Sub-step b3: If the number of points in the candidate plane is higher than the preset threshold for the number of points in the plane, calculate the average distance between these points in the candidate plane and the candidate plane, and proceed to the next plane sampling operation.

[0455] Specifically, in sub-step b3, once enough support points are found to confirm the existence of the candidate plane, the average distance between these points in the plane and the candidate plane can be determined as the mean distance error corresponding to the candidate plane. The smaller the mean distance error, the better and more reliable the fit, which can be used to evaluate the quality of the fitted plane.

[0456] Therefore, in sub-step b3, the current candidate plane, its corresponding set of points P_pre, and the average distance can be further recorded as the basis for subsequent processing. Furthermore, if the plane sampling operation in this iteration is successful, the iteration for the next plane sampling operation begins; that is, for the other 3D points among the selected n 3D points, sub-step b1 is executed again.

[0457] Sub-step b4: If the number of points in the candidate plane is not higher than the preset threshold for the number of points in the plane, the candidate plane fitted this time can be considered invalid, and the next plane sampling operation can be performed.

[0458] Specifically, in sub-step b4, it means that the fitted plane does not have enough support points, possibly because unsuitable points were randomly selected or the plane itself does not exist. In this case, the current plane should be abandoned, that is, the plane sampling operation of this iteration fails, and the next iteration of the plane sampling operation begins. That is, for other three-dimensional points among the selected n three-dimensional points, sub-step b1 is returned to be executed again.

[0459] In each round of planar detection, step b above will be executed n times. Specifically, for the n 3D points selected in this round of planar detection, after executing step b above, at most n average distances will be obtained, and then step c will begin:

[0460] Step c: Compare the at most n average distances determined in this round of planar detection operation, and determine the minimum average distance among them.

[0461] The phrase "at most n average distances" as described here means that the above n planar sampling operations can yield at most n average distances. Here, n equals n, indicating that all n planar sampling operations were successful, while less than n indicates that some planar sampling operations failed.

[0462] In step c, after obtaining the minimum average distance from the iterations of the above n planar sampling operations, step d is then executed:

[0463] Step d: Determine whether the minimum average distance is less than the preset error threshold.

[0464] Similar to the other threshold parameters mentioned above, the preset error threshold is also a distance-related value. This threshold can be set flexibly, and this article does not impose specific limitations on it.

[0465] In step d, if the judgment result is yes, then step e is executed; otherwise, step f is executed.

[0466] Step e: If the determined minimum average distance is less than the preset error threshold, the target plane is obtained by cleaning up the in-plane points of the target candidate plane corresponding to the minimum average distance.

[0467] In step e, if the determined minimum average distance is less than the preset error threshold, it indicates that the fitting result of this target candidate plane is reliable. It is considered that a better candidate plane has been found in the n iterations of the plane sampling operation. Therefore, based on the in-plane point set P_pre corresponding to the target candidate plane, in-plane point cleaning is performed to determine a target plane detected in this round of plane detection operation.

[0468] In this embodiment, in-plane point cleanup can be performed by clustering. One optional implementation includes sub-steps e1 to e6:

[0469] Sub-step e1: Project the points in each plane corresponding to the target candidate plane onto the target candidate plane to obtain the corresponding two-dimensional points.

[0470] Sub-step e2: Divide the target candidate plane into two sub-planes based on the distance between these two-dimensional points.

[0471] Specifically, in sub-steps e1 to e2, all in-plane points in P_pre corresponding to the target candidate plane are projected onto the target candidate plane to obtain two-dimensional points. The projection method is not specifically limited in this paper.

[0472] Then, based on the distance between the two-dimensional points, the k-means clustering algorithm (k=2) is used to quickly cluster these two-dimensional points to distinguish between points that belong to the target candidate plane and points that do not, ultimately resulting in two sub-planes.

[0473] In this embodiment, k=2 is chosen because of the speed requirement and because for the target candidate plane, it is only necessary to classify it into two categories: belonging to the target candidate plane and not belonging to the target candidate plane.

[0474] After dividing the target candidate plane into two sub-planes, the following sub-steps are performed:

[0475] Sub-step e3: Determine the average distance between a two-dimensional point in each subplane and the center of the corresponding subplane, and the center distance between the centers of two subplanes.

[0476] In this sub-step, for the two separated sub-planes, it is necessary to calculate the average distance from the center of the sub-plane and the interior point of each sub-plane to the center of the sub-plane, as well as the distance between the centers of the two sub-planes.

[0477] The center of a plane refers to the geometric center of all points within the plane, and is usually used to describe the position and orientation of the plane. The center of each subplane can be obtained by calculating the centroid of all points within the subplane, but this paper does not make any specific restrictions on this.

[0478] Assume that for the target candidate plane P, it is divided into sub-plane P1 and sub-plane P2. Among them, there are R1 two-dimensional points in sub-plane P1 and R2 two-dimensional points in sub-plane P2. Then in sub-step e3, for sub-plane P1, calculate the average value of the distances between these R1 two-dimensional points and the center of sub-plane P1, denoted as average distance D1. For sub-plane P2, calculate the average value of the distances between these R2 two-dimensional points and the center of sub-plane P2, denoted as average distance D2. In addition, also calculate the distance between the centers of the two sub-planes P1 and P2, denoted as center distance D3.

[0479] In the embodiments of the present application, by calculating these statistical data, the structure and characteristics of each sub-plane can be better understood, which helps to evaluate and verify whether the above two sub-planes indeed belong to the same plane, so as to ensure that the final plane detection result is as accurate as possible.

[0480] Sub-step e4: Determine whether the center distance is less than the target distance threshold.

[0481] Among them, the target distance threshold is positively correlated with the average distance corresponding to the larger plane in the two sub-planes.

[0482] In addition, in the embodiments of the present application, the size of the sub-plane is determined according to the number of two-dimensional points included in each sub-plane during the above clustering. Among them, the one with a larger number of included two-dimensional points is the larger plane, and the one with a smaller number of included two-dimensional points is the smaller plane. Of course, if the number of two-dimensional points included in the two sub-planes is the same, one of the two sub-planes can be randomly designated as the large plane.

[0483] [[ID=十六]]For example, the target distance threshold = the average distance corresponding to the larger plane * the threshold multiple.

[0484] In the embodiments of the present application, the threshold multiple can also be flexibly set, and no specific limitation is made herein, such as set to 3. Assume R1 > R2, then P1 is the larger plane, and the target distance threshold = 3D1; assume R1 < R2, then P2 is the larger plane, and the target distance threshold = 3D2; assume R1 = R2, then P1 or P2 can be made the larger plane. Correspondingly, the target distance threshold = 3D1 or 3D2.

[0485] If the center distance between the two sub-planes is less than the target distance threshold, it indicates that the two sub-planes are very close in space and may actually be different parts of the same plane. On the contrary, the two sub-planes are far apart in space and may actually be different planes.

[0486] Therefore, in sub-step e4, if the judgment result is yes, then execute sub-step e5, otherwise, execute sub-step e6:

[0487] Sub-step e5: If the center distance is less than the target distance threshold, then the candidate target plane is determined as the target plane.

[0488] In this sub-step, the center distance is less than the target distance threshold, so the two sub-planes are considered to be close and can be merged into one plane for processing. That is, the two sub-planes are different parts of the target candidate plane. In this case, the points P_pre in the plane do not need to be cleaned up, and the target candidate plane is directly used as a target plane successfully detected in this round of plane detection operation. In this case, the interior points in the target plane are the three-dimensional points in the set of interior points P_pre corresponding to the target candidate plane. This can ensure that the finally identified plane is more accurate, avoid incorrectly dividing points that should belong to the same plane into different planes, and ensure that adjacent plane segments can be correctly connected to form a larger plane region.

[0489] Sub-step e6: If the center distance is not less than the target distance threshold, then the larger plane is determined as the target plane.

[0490] In this sub-step, the center distance is not less than the target distance threshold, so the two sub-planes are considered to be far apart. In this case, the interior points corresponding to the smaller plane need to be cleared, and the larger plane among the two sub-planes is taken as a target plane successfully detected in this round of plane detection operation. The interior points in the target plane are the interior points of the larger plane, such as the three-dimensional points corresponding to the two-dimensional points in the larger plane.

[0491] like Figure 11 As shown, this is a schematic diagram of in-plane point clearing in an embodiment of this application. Wherein, Figure 11 This means that the two-dimensional points obtained by projecting the points in the plane corresponding to the target candidate plane are clustered to obtain two sub-planes. The center distance between these two sub-planes is greater than the average distance of the larger sub-plane multiplied by a threshold multiple. Therefore, the two sub-planes are considered to be far apart, so only the larger plane is selected, the smaller plane is cleaned up, and the interior points are updated to the interior points of the larger plane.

[0492] For the newly obtained target plane, its interior point set has also been determined. Based on this, the plane equation and normal of the target plane are solved again using the above formula 8 based on the least squares method, and the outer contour of the plane and the transformation from the world coordinate system to the plane coordinate system are calculated. Thus, all the parameters of the target plane are obtained, which can be used for the interactive placement of virtual objects.

[0493] In the embodiments of this application, a more accurate plane equation can be obtained through the above-described quadratic fitting process.

[0494] If the maximum number of iterations (m) is reached, the plane detection step is exited; otherwise, the next plane detection step is initiated.

[0495] In the above implementation, computational complexity can be reduced by random sampling and nearest neighbor search. The validity of the plane hypothesis can be verified by statistically minimizing the mean distance error. Noise points can be effectively removed by in-plane point cleaning and clustering. By calculating the plane center and the average distance from the in-plane points to the center, the characteristics of each plane can be better understood and characterized. Then, by determining whether to merge two sub-planes, the problem of over-segmentation can be reduced and the overall quality of the detection results can be improved. This method not only helps to reduce misclassification, but also ensures that the final plane detection results are more accurate and reliable, which is crucial for applications such as augmented reality.

[0496] In summary, the method iterates through multiple iterations to find the most suitable plane from the 3D point cloud, thereby improving the accuracy and robustness of plane detection. Finally, objects can be interactively placed as AR virtual objects based on the detected planes, resulting in a better object experience.

[0497] Step f: If the determined minimum average distance is not less than the preset error threshold, then it is determined that no plane can be detected in this round of plane detection operation, and the iteration of the plane detection operation ends.

[0498] In step f, if the determined minimum average distance is not less than the preset error threshold, it indicates that the plane fitting result is unreliable, meaning that there is no plane in the scene. At this point, regardless of whether the maximum m iterations have been reached, the plane detection step is exited, and there is no need to execute the remaining iteration process of the plane detection operation within m iterations.

[0499] In summary, steps a to f above illustrate the implementation process of a planar detection operation in this application.

[0500] During the initial reconstruction process, multiple target planes, denoted as plane_n, can be obtained from the existing map by iterating through up to m rounds of plane detection operations. At this point, both the initial reconstruction result and the plane result have been obtained. The detected plane result (plane_n) can be used for augmented reality applications. For example, objects can now be placed as AR virtual objects in the real environment, ensuring that they are stably attached to the detected target planes, thus satisfying mobile interaction more quickly.

[0501] Of course, besides the plane detection methods mentioned above, other methods can also be used, such as the RANSAC algorithm, region growing, or segmentation-based methods. The RANSAC algorithm estimates the model through random sampling and optimizes the plane by evaluating interior points. It can handle point cloud data with a lot of noise, has high computational efficiency, and is especially suitable for large point cloud datasets. The region growing method expands neighboring points based on seed points until a stopping condition is met. This method can discover large-area planar regions with similar properties, and the results are usually more refined than RANSAC. Segmentation-based methods typically include several stages: preprocessing, feature extraction, clustering, and post-processing. They can handle complex scenes, identify multiple planes, and provide more geometric information, which is helpful for subsequent AR applications. Of course, this is just a simple example. Other plane detection methods are also applicable to the embodiments of this application. In practical applications, the choice of which plane detection method to use depends on specific needs and available resources. These methods can also be combined to achieve the best results, and this document does not impose specific limitations on this.

[0502] The above is an introduction to the initial reconstruction operation. For a summary, please refer to... Figure 12 As shown, it is a logic block diagram of an initial reconstruction method in an embodiment of this application.

[0503] For the video to be processed, firstly, based on the keyframe selection method described above, the first and second keyframes are selected. Pose estimation and two-frame triangulation are then performed on the first and second keyframes. Based on this, pose estimation and two-frame triangulation are performed frame-by-frame on the other initial video frames in between. Further, multi-frame triangulation and global basis comparison (BA) are performed on the first to second keyframes. If the first keyframe is not the first frame in the video, pose estimation and two-frame triangulation are then performed on its preceding frames. Furthermore, up to m plane detections can be performed; for specific implementation details, please refer to the above embodiments, and repeated details will not be elaborated further.

[0504] The initial reconstruction operation described above completes the initial mapping optimization and initial plane detection. Simultaneously, the feature matching module of the matching thread is also processing point tracking and matching for subsequent frames. Therefore, after the initial reconstruction and plane detection, the system can continue to perform incremental reconstruction, that is, continuously updating and refining the 3D map and planar information as more frames are processed. The implementation method for iteratively executing incremental reconstruction operations is described below:

[0505] S422: After the initial reconstruction operation is completed, the incremental reconstruction operation is iteratively executed in the following manner until the video processing is completed: Based on the number of matching feature points and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed for each incremental video frame; wherein, in the subsequent iteration process except for the first round of incremental reconstruction operation, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

[0506] Specifically, when iteratively executing incremental reconstruction operations, the number of executions depends on the actual situation such as the length of the video and the video content itself. Assuming a total of P executions, the second and third keyframes are essentially variables in these P rounds of incremental reconstruction operations.

[0507] For example, in the initial reconstruction operation, the first keyframe selected is frame 1 of the video, and the second keyframe is frame 10. Then, in the first round of incremental reconstruction, the second keyframe is frame 10, and the third keyframe selected after frame 10 is, say, frame 20. In the second round of incremental reconstruction, the second keyframe should be frame 20, and the third keyframe selected after frame 20 is, say, frame 30. In the third round of incremental reconstruction, the second keyframe should be frame 30, and the third keyframe selected after frame 30 is, say, frame 40; and so on. Of course, in the final round of incremental reconstruction, the last frame of the video can be directly used as the third keyframe for that round.

[0508] In the reconstruction thread of this application, in addition to the initial reconstruction based on the above method, the incremental reconstruction operation will continue to be performed iteratively using the points of new frames. As more frames are added and more feature points are matched, the pose information and 3D point cloud of the entire system will gradually become more accurate and complete, thereby satisfying the effect of full video AR.

[0509] Specifically, incremental reconstruction is a dynamic process designed to update and refine the 3D reconstruction structure in real time as more data becomes available.

[0510] Similar to the initial reconstruction operation described above, when performing incremental reconstruction operation in each iteration, it is necessary to first select key frames. In order to optimize computational efficiency and maintain good reconstruction quality, starting from the j-th frame (the current second key frame), the k-th frame (the third key frame) is selected by detecting the number of feature points and the average optical flow displacement.

[0511] In one alternative implementation, by means of... Figure 13 The process shown selects the third keyframe for this round of incremental reconstruction, including the following steps S131 to S134:

[0512] S131: Take the current second keyframe as the j-th frame, and take the first frame after the second keyframe in the video as the initial k-th frame.

[0513] In this step, if the current incremental reconstruction operation is the first initial reconstruction operation after the initial reconstruction operation, then j is the second keyframe selected in the initial reconstruction operation. For example, if i=1 and j=10 were selected in the initial reconstruction, then j=10 in this incremental reconstruction operation. Based on the latest j, let the initial k=j+1, then k=11. As another example, if i=2 and j=20 were selected in the initial reconstruction, then j=20 in this incremental reconstruction operation, and the initial k=21; and so on.

[0514] In this step, if the current incremental reconstruction operation is not the first initial reconstruction operation after the initial reconstruction operation, then j is the third keyframe selected in the previous incremental reconstruction operation. For example, if j=20 and k=30 in the previous incremental reconstruction operation, then in the current incremental reconstruction operation, j=30 and the initial k=31; and so on.

[0515] Here, "initial" means that k may change subsequently, as detailed below:

[0516] S132: For the j-th frame and the current k-th frame, determine whether the number of each matching feature point and the inter-frame displacement satisfy the selection conditions.

[0517] Specifically, if the selection criteria are not met, then execute S133; if the selection criteria are met, then execute S134.

[0518] The selection criteria are as follows: the number of matched feature points is greater than the third quantity threshold, and the average optical flow displacement of the matched feature points is greater than the preset displacement threshold, as described above. Figure 10 Some related descriptions will not be repeated here.

[0519] S133: If the selection condition is not met, let k = k + 1 and return to S132.

[0520] Specifically, if the selection conditions are not met between the current j-th frame and the k-th frame, it means that (j, k) at this time either does not have enough matching feature points or does not have significant motion changes. Therefore, this step is performed: slide k, let k = k + 1, and continue to try subsequent frames until a frame pair that meets the conditions is found, or until the current k-th frame is the latest / last frame in the video.

[0521] Conversely, if the above selection conditions are met between the current j-th frame and the k-th frame, it means that there are enough matching feature points at (j, k) and there is a significant motion change, so step S134 is executed:

[0522] S134: If the selection criteria are met, the newly determined k-th frame will be used as the third keyframe.

[0523] In this step, simply select the current frame pair and use the current k-th frame as the third keyframe for this round of incremental operations. It should also be noted that in the final round of incremental reconstruction, the last frame of the video can be directly used as the third keyframe for that round.

[0524] In the embodiments of this application, the above Figure 13 The described process and Figure 10 The difference lies in the process described. Figure 10 The 'i' in the equation can slide, while Figure 13 In this context, j is fixed, and in the next round of incremental reconstruction, the third keyframe k obtained from the previous round of incremental reconstruction will be used as the latest second keyframe j. This is to ensure that no frames are missing during the incremental reconstruction process, so that every frame after j is processed, thereby improving the accuracy of the 3D reconstruction results.

[0525] In the above implementation, for subsequent frames after frame j, keyframe selection is performed first. This reduces computational load while improving the success rate of incremental reconstruction. Specifically, starting from frame j, and as described above... Figure 10 Similar to selecting keyframes, the number of feature points from frame j to the subsequent frame k and the average optical flow displacement are counted. When the number of remaining feature points and the optical flow displacement are sufficient, frame k is selected. This allows for the selection of a suitable third keyframe so that the pose can be successfully recovered to construct the camera's pose map and track its motion in the environment.

[0526] Specifically, during each round of incremental reconstruction, after selecting the third keyframe k, pose estimation is performed on frame k, and then new 3D points are solved through triangulation. The specific process is as follows:

[0527] An optional implementation involves determining the pose information corresponding to each incremental video frame within the range of the second to the third keyframes in the following manner:

[0528] Based on the pose information corresponding to the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional and two-dimensional points, the pose information corresponding to each incremental video frame in the range from the second keyframe to the third keyframe is determined frame by frame.

[0529] In this embodiment, since a point cloud map containing several 3D points has been obtained based on the previous initial reconstruction operation, or even the previous rounds of incremental reconstruction operation, and the map is known here, the pose of all frames from j to k is optimized. For each frame between frame j and frame k, the pose can be recovered using the above-mentioned module 1. In summary, the initial pose of the current frame is first estimated using the uniform motion assumption, and then the known 3D point cloud and the 2D feature points of the current frame are matched to optimize the pose information corresponding to the current frame by minimizing the projection error. For specific implementation details, please refer to the relevant description in the above-mentioned module 1 section, which will not be repeated here.

[0530] Based on the above implementation method, the pose information of each incremental video frame between the second keyframe and the third keyframe can be determined frame by frame. The entire process combines the simplicity of the uniform motion assumption with the accuracy of nonlinear optimization based on 3D-2D projection. In this way, the pose change of the camera between adjacent frames can be effectively estimated.

[0531] Through the above process, the pose information corresponding to all frames from j to k can be obtained. However, in the incremental reconstruction operation, the purpose is to perform 3D reconstruction. Therefore, in the process of calculating the pose information, triangulation processing is also needed to recover several 3D points, thereby gradually enriching the map. An optional implementation method is as follows:

[0532] First, for each incremental video frame within the range from the second keyframe to the third keyframe, after determining the corresponding pose information frame by frame by combining the uniform motion model and the projection constraints of the three-dimensional point and the two-dimensional point, the pose information corresponding to this incremental video frame and the previous video frame, combined with the position information of each matching feature point between this incremental video frame and the previous video frame, can be used to perform two-frame triangulation processing on each matching feature point between this incremental video frame and the previous video frame.

[0533] Subsequently, based on the pose information corresponding to each incremental video frame within the range of the second to the third keyframe, and combined with the position information of each matching feature point between each incremental video frame, multi-frame triangulation processing is performed on each matching feature point between each incremental video frame.

[0534] In this embodiment of the application, for each frame between frame j and frame k, while restoring the pose using the above-mentioned module 1, it is necessary to establish new observation 3D feature points. Specifically, new 3D points can be obtained by triangulation of two frames using module 2.

[0535] For example, using module 1, after determining the pose of frame j+1 based on the poses of frame j and frame j-1, the matching feature points in frame j and frame j+1 can be triangulated. Using module 2, multiple 3D points are recovered. These are points jointly observed in frame j and frame j+1. These 3D points form a 3D point cloud shared by frames j and j+1. This point cloud describes the true position of objects or scene elements in the scene that are jointly observed by the images in frames j and j+1.

[0536] Furthermore, using module1, the pose of frame j+2 can be determined based on the poses of frame j and frame j+1. Then, the matching feature points in frame j+1 and frame j+2 can be triangulated into two frames. Using module2, multiple 3D points can be recovered. These point clouds describe the real positions of objects or scene elements in the scene that are co-viewed by the images of frames j+1 and j+2.

[0537] By following this logic, all poses and new 3D points from frame j to frame k can be obtained.

[0538] Since the above process involves triangulation between two frames, it is prone to failure. Subsequently, module 3 can be used to perform multi-frame triangulation to further supplement 3D points. For specific implementation details, please refer to the above embodiments, which will not be repeated here.

[0539] Specifically, for each feature point, M frames are observed, and the common 2D points of the M observation frames are collected and projected to form 2M linear equations. The coordinates of the 3D point are then solved. In each incremental reconstruction operation, the value of M represents the summary of each initial video frame within the range of the second keyframe (frame j) to the third keyframe (frame k). For example, if j = 20 and k = 30, then M = 11, and 22 linear equations can be constructed based on Formula 6 to perform multi-frame triangulation processing on the common matching feature points between frames 20 and 30. Or, if j = 30 and k = 45, then M = 16, and 32 linear equations can be constructed based on Formula 6 to perform multi-frame triangulation processing on the common matching feature points between frames 30 and 45; and so on.

[0540] In addition, after performing multi-frame triangulation on frames j to k, the incremental reconstruction result of this round of incremental reconstruction operation can be obtained. Based on this, similar to the above process, abnormal 3D points can also be deleted by at least one of the above anomaly check two and anomaly check three.

[0541] Based on the above implementation method, reconstruction begins when the number of processed feature points and the length of the optical flow trajectory exceed the threshold. An adaptive key frame selection strategy is adopted, and by adding temporal constraints, the pose and spatial 3D points of each frame are quickly restored.

[0542] To further improve the accuracy of incremental 3D reconstruction results, after performing multi-frame triangulation on frames j to k during each incremental reconstruction operation, an optional implementation is to perform incremental BA optimization on the current incremental 3D reconstruction results.

[0543] In this embodiment, incremental BA can also be understood as local BA, which is a process of optimization based on new data within a specific time period, using a known initial map. It only optimizes the most recent frame and related 3D points without changing the pose of historical frames (i.e., "old frames"). However, if these historical frames have observations of 3D points involved in the most recent frame, then this observation information will still be used as constraints to help optimize the position of the current frame and 3D points.

[0544] Based on the above ideas, the optimization terms for incremental BA optimization in this application are: pose information, camera intrinsic parameters K, and 3D points observed during the period from the second keyframe (the current j-th frame) to the third keyframe (the current k-th frame). The constraint terms include the above formula 4 and pose continuity constraints (i.e., adjacent frame constraints).

[0545] Note the difference between incremental BA and global BA mentioned above: for frames before j, no optimization is performed. However, if older frames observe 3D points between j and k, these observations are still included in the constraints, but the pose of the older frames is not optimized. In other words, the pose of frames before j remains fixed, but observations of 3D points between j and k still need to be considered.

[0546] The phrase "old frames have observations of 3D points between j and k" means that in the time series, those frames (i.e., old frames) that are before frames j and k have visibility or observations of the reconstructed 3D points between frames j and k.

[0547] like Figure 14 As shown, this is a schematic diagram of an incremental BA optimization item in an embodiment of this application. The solid triangles represent video frames that need optimization in incremental BA, the solid squares represent map points (i.e., 3D points in this document) that need optimization in incremental BA, and the dashed triangles represent video frames that do not need optimization in incremental BA, i.e., historical video frames. Figure 14 The optimization process of incremental BA is demonstrated. During optimization, historical video frames are fixed, and only the video frames and the 3D points between them are optimized. If 3D points between the optimized video frames are observed in historical video frames, constraints are added, but the pose of the historical video frames is not optimized.

[0548] After optimization, feature point anomaly removal is performed, the projection error of each 3D point is checked, points with excessive errors are removed, and depth values ​​are checked to remove negative values ​​and far points.

[0549] In the above implementation, although the poses of the old frames are fixed, their observation information on the 3D points in the most recent frames still serves as a constraint, helping to optimize the position of these points and the pose of the most recent frames. This can improve the accuracy and reliability of 3D reconstruction, reduce the impact of noise and outliers, and obtain higher quality point clouds.

[0550] Incremental plane detection is performed after incremental BA. Specifically, if there is no plane now (i.e. no suitable target plane was detected in the previous initial reconstruction operation or the previous round of incremental reconstruction operation), then in the current round of incremental reconstruction operation, at most m plane detection operations are performed on the point cloud that does not currently belong to any plane. Each round of plane detection operation performs the following process steps a to f to obtain several planes.

[0551] If there are planes now (i.e., some suitable target planes were detected in the previous initial reconstruction operation or the previous round of incremental reconstruction operation), then in the current round of incremental reconstruction operation, the point cloud that does not currently belong to any plane is iteratively executed for up to m rounds of plane detection operation. Each round of plane detection operation executes the following process steps a to f to obtain several planes, and then plane merging is performed.

[0552] The execution process of the plane detection operation is detailed in the above embodiments and will not be repeated here.

[0553] The process of merging planes is explained in detail below:

[0554] An alternative implementation involves merging planes during incremental reconstruction, after iteratively performing plane detection operations, in the following manner:

[0555] For each target plane that has been obtained, the target planes that meet the preset plane merging conditions are merged based on the normal vector, outer contour polygon, and outer contour polygon after mutual projection of each pair of target planes.

[0556] The plane merging condition is set according to actual application requirements. In this embodiment, it can be set as follows: a. The angle between the normal vectors of the two target planes is less than the angle threshold t. angle b. The polygon obtained by projecting the outer contour polygon of target plane 1 onto another target plane 2 has an overlap area with the outer contour polygon of target plane 2 that is greater than the area threshold t2, and the area intersection-union ratio is greater than the intersection-union ratio threshold t3. The polygon projected from target plane 2 onto target plane 1 also needs to meet the above conditions.

[0557] If the above conditions are met, a planar merge will be performed. The merge method is as follows:

[0558] After merging all interior points of the two target planes to form a new set of plane points, a single plane is fitted based on this new set of plane points, and plane-related parameters such as plane equations and normals, plane outer contours, and transformations from the world coordinate system to the plane coordinate system are updated for object interaction and placement of virtual objects.

[0559] The specific implementation method for fitting a single plane based on the new plane point set can be found in the relevant description of Formula 8 above. In Formula 8, (x, y, z) are the 3D coordinates of each point in the new plane point set. Based on Formula 8, the most suitable plane parameters A, B, C, and D can be found by using the least squares method to minimize the sum of the squares of the distances from all points in the new plane point set to this plane. The plane determined by these plane parameters is the plane after merging the two target planes in this embodiment of the application.

[0560] In practical applications, all detected target planes in the current video can be traversed, and it can be determined whether the plane information of any two target planes traversed meets the above plane merging conditions. If they do not meet the conditions, they cannot be merged; if they do meet the conditions, that is, the two target planes belong to one plane, they can be merged. This process is repeated until all planes that meet the above plane merging conditions are merged, so as to obtain all merged target planes and the remaining target planes that cannot be merged.

[0561] like Figure 15 As shown, this is a schematic diagram of a plane merging process in an embodiment of this application, illustrating the merging of two target planes on the left to obtain a larger plane on the right. The merging method involves combining all point sets of the two target planes into a new set of plane points, and then performing plane fitting as described above to ultimately obtain a unified larger plane.

[0562] Based on the above implementation method, plane merging during 3D reconstruction helps improve reconstruction accuracy and model simplicity. It eliminates duplicate and flat areas, reduces redundant data, and lowers computational complexity. Furthermore, the merged single plane is easier to understand and analyze, facilitating subsequent scene understanding and application.

[0563] The above is an introduction to a round of incremental reconstruction operations. For a summary, please refer to... Figure 16 As shown, it is a logic block diagram of an incremental reconstruction method in an embodiment of this application.

[0564] For the video to be processed, firstly, based on the keyframe selection method described above, a third keyframe is selected after the second keyframe. Then, pose estimation and two-frame triangulation are performed frame-by-frame on each incremental video frame between the second and third keyframes. Further, multi-frame triangulation and incremental BA are performed on the second to third keyframes. Based on this, up to m plane detections and plane merging can be performed. For specific implementation details, please refer to the above embodiments; repeated details will not be elaborated further.

[0565] When performing each round of incremental reconstruction, the incremental 3D reconstruction result obtained in that round is: the 3D point cloud obtained through two-frame triangulation and multi-frame triangulation in that round, and may also include some planes in the video scene. This incremental 3D reconstruction result is integrated with the initial 3D reconstruction result obtained previously, as well as the incremental 3D reconstruction results from previous rounds, which may also exist. This is the result of 3D reconstruction of the video in this application embodiment. As more frames are added and more feature points are matched, the result will gradually become more accurate and complete, thereby satisfying the effect of full-video AR.

[0566] In one alternative implementation, after the initial reconstruction operation and incremental reconstruction operation are completed, global BA optimization can be performed on the currently obtained 3D reconstruction results.

[0567] The optimization terms for this global BA are: pose of all video frames, camera intrinsics, all 3D points, and planar parameters. Constraints include: the error of the 2D projection of 3D points to the current frame, with the projection equation being Equation 4 above; and constraints on points within the plane, corresponding to Equation 8 above.

[0568] The projection equation in the constraint terms describes the relationship between 2D feature points and their corresponding 3D points. For each pair of 2D-3D correspondences, this relationship is represented by Equation 4 above. The goal of the projection equation is to minimize the difference between the actually detected 2D feature points and the predicted 2D feature points, i.e., the projection error. This constraint helps ensure that the optimized 3D points and camera pose accurately reflect the geometric relationships of the real world. Accuracy is improved by minimizing the projection error and utilizing prior knowledge to limit the solution space.

[0569] The constraint on points within the plane ensures that the 3D points fall on the known planes, enhancing the consistency of the reconstruction results. These planes are extracted from previous steps, and for each 3D point considered to be located on a plane, Equation 8 above is satisfied. This constraint helps maintain the consistency between the 3D points and the planes, thus avoiding inconsistencies caused by noise or incorrect matching.

[0570] In the above implementation, by combining these two constraints, global BA can simultaneously consider the observation accuracy of 2D features and the consistency of planar structure during the optimization process, thereby producing more accurate 3D reconstruction results.

[0571] After optimization, the video reconstruction and planar generation results contain detailed scene information, which can support subsequent interactive applications, such as the placement of virtual objects.

[0572] In short, see Figure 17 The diagram shown is a logical block diagram of a video 3D reconstruction method according to an embodiment of this application. This method aims to solve the problems of scene reconstruction and plane detection on mobile devices, in order to achieve AR placement effects on mobile devices, such as advertising wall placement. The solution is implemented through the following steps:

[0573] First, the input video passes through a matching thread, which extracts feature points sequentially and performs optical flow tracking frame by frame to obtain point pairing information, namely the inter-frame feature point association information and the position information of each matched feature point in consecutive frames, as mentioned earlier. Next, the reconstruction process begins, executed by a reconstruction thread. Initial reconstruction is performed by dynamically selecting two frames (the first and second keyframes) to recover the relative pose. PnP (module 1), two-frame triangulation (module 2), multi-frame triangulation (module 3), global BA, and restoration of preceding frames are used to obtain the camera pose and 3D point cloud of the second keyframe and preceding frames in the video. Simultaneously, incremental reconstruction is performed, iteratively optimizing each time. Each time, a subsequent frame (the third keyframe) is selected, and PnP (module 1), two-frame triangulation (module 2), multi-frame triangulation (module 3), and local BA are used to obtain the camera pose and 3D point cloud within the range from the second to the third keyframe in the video. Simultaneously, incremental plane fitting is performed, using multiple 3D point RANSAC fittings to the plane, updating and optimizing the plane parameters. Finally, global basis analysis (BA) is used to further optimize the pose, 3D point cloud, and planar parameters of all frames to obtain high-quality reconstruction results. For specific implementation details, please refer to the above embodiments; repeated details will not be elaborated further.

[0574] The advantage of this method lies in its lightweight feature matching module and multi-threaded incremental optimization mechanism, which effectively reduces user waiting time and improves interaction performance when running on mobile devices. In this way, users can obtain a faster and smoother AR experience, especially in scenarios such as placing text effects, where they can perceive the position and posture of objects in the real environment in real time, thus enhancing the user experience.

[0575] The entire process described above is designed to construct a dense 3D environment model from a monocular video sequence while simultaneously estimating the camera pose for real-time interaction in AR applications. This process, by combining multiple optimization strategies and depth deletion, ensures the accuracy and robustness of the reconstruction results, particularly when handling scenes with large depth of field and low texture. Furthermore, the process incorporates incremental and global optimization strategies, effectively processing consecutive video frames, continuously refining and expanding the 3D environment model, and identifying and updating planar information, making AR applications more accurate and smoother, especially when running on mobile devices.

[0576] It should be emphasized that this application allows for flexible adjustment of various thresholds according to actual conditions to adapt to the needs of different scenarios, such as different types of video content or different computing resource limitations.

[0577] Based on the same inventive concept, embodiments of this application also provide a video-based 3D reconstruction device. For example... Figure 18 As shown, it is a structural schematic diagram of a video-based 3D reconstruction device 1800 according to an embodiment of this application, which may include:

[0578] The feature processing unit 1801 is used to detect, frame by frame, each matching feature point between frames and the position information of each matching feature point on the corresponding video frame for the video to be processed.

[0579] The reconstruction unit 1802 is configured to perform the following reconstruction operation based on the detected video frames during the frame-by-frame detection process of the feature processing unit, and the feature processing unit is configured to continue processing undetected video frames:

[0580] The initial reconstruction operation is performed in the following manner: based on the number of each matching feature point and the inter-frame displacement, a first keyframe and a second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame.

[0581] The incremental reconstruction operation is performed iteratively in the following manner until the video processing is completed: based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed on each incremental video frame; wherein, in subsequent iterations other than the first round of incremental reconstruction operation, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

[0582] Optionally, the feature processing unit 1801 is specifically used for:

[0583] For each video frame in the video, perform the following operations in sequence:

[0584] If the number of feature points in the preceding video frame adjacent to the video frame is greater than a first number threshold, then optical flow tracing is used to determine each matching feature point between the video frame and the preceding video frame, as well as the position information of each matching feature point on the video frame and the preceding video frame.

[0585] If the number of feature points in the preceding video frame adjacent to the video frame is not greater than a first number threshold, then each feature point on the video frame is extracted, and each matching feature point between the video frame and the preceding video frame is determined by optical flow tracing, as well as the position information of each matching feature point on the video frame and the preceding video frame.

[0586] Optionally, the feature processing unit 1801 is specifically used for:

[0587] At least one instruction set acceleration method is used to perform accelerated segment test feature extraction on the video frame to obtain each feature point contained in the video frame and the response value of each feature point;

[0588] The video frame is divided into multiple filtering grids;

[0589] For each filter grid, the feature points in the filter grid are filtered according to the response value of each feature point in the filter grid.

[0590] Optionally, after filtering the feature points in the filter grid according to the response value of each feature point in the filter grid for each filter grid, the feature processing unit 1801 is further configured to:

[0591] If the number of feature points in the video frame is greater than the second number threshold, then at least one feature point with line features is deleted based on the difference between the gradient direction angles of each feature point in the video frame.

[0592] For each feature point in the video frame, whether the feature point has line features is determined in the following way:

[0593] According to the preset range parameters, the detection area is determined with the feature point as the center point;

[0594] The gradient direction angle of each feature point in the region to be detected is determined, as well as the gradient interval to which each gradient direction angle belongs; wherein each gradient interval is obtained by dividing the range of gradient direction angle values ​​at equal intervals;

[0595] Determine the target gradient interval containing the most feature points in each gradient interval, and the reference gradient interval containing the second most feature points.

[0596] If the target gradient interval contains the center point, and the angle difference between the center point and the reference gradient interval exceeds a preset angle threshold, then the center point is determined to have line characteristics.

[0597] Optionally, when detecting each matching feature point between frames and the position information of each matching feature point on the corresponding video frame, the feature processing unit 1801 is further configured to:

[0598] For each pair of adjacent frames in the video, perform at least one of the following verification operations:

[0599] By performing random sampling consistency verification on each matching feature point in two adjacent frames, matching feature points that fail the verification are deleted.

[0600] The preceding video frame in two adjacent frames is divided into multiple verification grids. For each matching feature point in each verification grid: if the difference between the optical flow displacement of the matching feature point between two adjacent frames and the corresponding average optical flow displacement is greater than a preset difference threshold, the matching feature point is deleted. The average optical flow displacement is the average value of the optical flow displacement of each matching feature point in the verification grid to which the matching feature point belongs between two adjacent frames.

[0601] Optionally, the reconstruction unit 1802 is specifically used for:

[0602] The first frame of the video is taken as the initial i-th frame, and the second frame of the video is taken as the initial j-th frame;

[0603] For the current i-th frame and j-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold.

[0604] If the selection conditions are not met, let j = j + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame;

[0605] If the selection conditions are met, pose estimation is performed on the current i-th frame and j-th frame, and based on the pose estimation results, the latest determined i-th frame is taken as the first keyframe, and the latest determined j-th frame is taken as the second keyframe.

[0606] Optionally, the reconstruction unit 1802 is specifically used for:

[0607] For the current i-th and j-th frames, determine the matching inliers between frames based on the pose estimation results;

[0608] If the total number of all matched points is greater than the fourth number threshold, then the current i-th frame is taken as the first keyframe and the current j-th frame is taken as the second keyframe.

[0609] If the total number of the matched points is not greater than the fourth quantity threshold and the total number of the matched feature points is not greater than the fifth quantity threshold, then let i = i + 1, and return to the step of analyzing whether the number of the matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame.

[0610] If the total number of all matched intrapoints is not greater than the fourth quantity threshold, and the total number of all matched feature points is greater than the fifth quantity threshold, then the preset displacement threshold is increased, j = j + 1, and the process of analyzing whether the number of all matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame is returned.

[0611] Optionally, the reconstruction unit 1802 is specifically used for:

[0612] Take the current second keyframe as the j-th frame, and take the first frame after the second keyframe as the initial k-th frame;

[0613] For the j-th frame and the current k-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold.

[0614] If the selection conditions are not met, let k = k + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the j-th frame and the current k-th frame;

[0615] If the selection criteria are met, the newly determined k-th frame will be used as the third keyframe.

[0616] Optionally, the reconstruction unit 1802 is further configured to:

[0617] The pose information corresponding to each initial video frame within the range from the first keyframe to the second keyframe is determined in the following manner:

[0618] Based on the camera intrinsic parameters and the position information of each matching feature point between the first keyframe and the second keyframe, the relative pose between the first keyframe and the second keyframe is determined.

[0619] Based on the relative pose, determine the pose information corresponding to the first keyframe and the second keyframe when the first keyframe is used as the reference coordinate system;

[0620] Based on the pose information corresponding to the first keyframe and the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to other initial video frames between the first keyframe and the second keyframe is determined frame by frame.

[0621] Furthermore, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined in the following manner:

[0622] Based on the pose information corresponding to the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined frame by frame.

[0623] Optionally, the reconstruction unit 1802 is specifically used for:

[0624] Based on the pose information corresponding to the first keyframe and the second keyframe, and combined with the position information of each matching feature point between the first keyframe and the second keyframe, two-frame triangulation processing is performed on each matching feature point between the first keyframe and the second keyframe.

[0625] For other initial video frames between the first keyframe and the second keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other initial video frames and their previous video frames, and combined with the position information of each matching feature point between the other initial video frames and their previous video frames, two-frame triangulation processing is performed on each matching feature point between the other initial video frames and their previous video frames.

[0626] The 3D point cloud obtained through two frames of triangulation during the initial reconstruction operation will be used as the initial 3D reconstruction result.

[0627] Optionally, the reconstruction unit 1802 is further configured to:

[0628] Perform at least one of the following optimization operations on the initial 3D reconstruction results:

[0629] Based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, and combined with the position information of each matching feature point between each initial video frame, multi-frame triangulation processing is performed on each matching feature point between each initial video frame; the three-dimensional point cloud obtained through multi-frame triangulation processing is added to the initial three-dimensional reconstruction result to optimize the initial three-dimensional reconstruction result.

[0630] Perform global BA optimization on the current initial 3D reconstruction results.

[0631] Optionally, if the first keyframe is not the first frame in the video, then during the initial reconstruction operation, the reconstruction unit 1802 is further configured to:

[0632] For the other video frames before the first keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other video frames and the next video frame, and combined with the position information of each matching feature point between the other video frames and the next video frame, two-frame triangulation processing is performed on each matching feature point between the other video frames and the next video frame.

[0633] The 3D point cloud obtained through two frames of triangulation is added to the current initial 3D reconstruction result to optimize the initial 3D reconstruction result.

[0634] Optionally, the reconstruction unit 1802 is specifically used for:

[0635] For each incremental video frame within the range from the second keyframe to the third keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the incremental video frame and its previous video frame, and combined with the position information of each matching feature point between the incremental video frame and its previous video frame, two-frame triangulation processing is performed on each matching feature point between the incremental video frame and its previous video frame.

[0636] Based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, and combined with the position information of each matching feature point between each incremental video frame, multi-frame triangulation is performed on each matching feature point between each incremental video frame.

[0637] The 3D point cloud obtained through two-frame triangulation and multi-frame triangulation during this round of incremental reconstruction will be used as the result of this round of incremental 3D reconstruction.

[0638] Incremental BA optimization was performed on the incremental 3D reconstruction results of this round.

[0639] Optionally, after obtaining the 3D reconstruction result containing the 3D point cloud, the reconstruction unit 1802 is further configured to:

[0640] The following plane detection operation is performed iteratively until a preset number of iterations is reached, or no plane is detected in the latest round of plane detection operation:

[0641] From the 3D point cloud, select multiple 3D points that are currently not assigned to a plane. For each 3D point, perform the following operations: fit a candidate plane based on a preset number of nearest neighbors of the 3D point in the 3D point cloud; if the number of points in the candidate plane is higher than a preset threshold for the number of points in the plane, determine the average distance between the points in the plane and the candidate plane; wherein, the points in the plane are the 3D points among the currently unassigned 3D points that are less than a preset distance threshold from the candidate plane.

[0642] Compare the various average distances determined in this round of planar detection operations to determine the minimum average distance;

[0643] If the determined minimum average distance is not less than the preset error threshold, then it is determined that the plane cannot be detected in this round of plane detection operation;

[0644] If the determined minimum average distance is less than the preset error threshold, the target plane is obtained by cleaning up the in-plane points of the target candidate plane corresponding to the minimum average distance.

[0645] Optionally, the reconstruction unit 1802 is specifically used for:

[0646] Project the points in each plane corresponding to the target candidate plane onto the target candidate plane to obtain the corresponding two-dimensional points;

[0647] Based on the distance between two-dimensional points, the target candidate plane is divided into two sub-planes;

[0648] Determine the average distance between a two-dimensional point in each subplane and the center of the subplane, and the center distance between the centers of two subplanes;

[0649] If the center distance is less than the target distance threshold, then the target candidate plane is determined as the target plane; the target distance threshold is positively correlated with the average distance corresponding to the larger of the two sub-planes;

[0650] If the center distance is not less than the target distance threshold, then the larger plane is determined to be the target plane.

[0651] Optionally, during the incremental reconstruction operation, after iteratively performing the plane detection operation, the reconstruction unit 1802 is further configured to:

[0652] For each target plane that has been obtained, the target planes that meet the preset plane merging conditions are merged based on the normal vector, outer contour polygon, and outer contour polygon after mutual projection of each pair of target planes.

[0653] Optionally, after the initial reconstruction operation and the incremental reconstruction operation are completed, the reconstruction unit 1802 is further configured to:

[0654] Perform global BA optimization on the currently obtained 3D reconstruction results.

[0655] Specifically, this application performs feature point matching frame by frame to determine the association information of feature points between frames and the position information of each matched feature point in consecutive frames, which is used to provide stable and effective inter-frame feature point tracking pairs for subsequent 3D reconstruction. In this process, after performing feature point matching on some frames in the video, the reconstruction operation can be started based on the detected video frames, while continuing to detect subsequent undetected video frames frame by frame.

[0656] In the initial reconstruction operation, keyframes are adaptively selected based on the number and displacement of matching feature points, thereby performing initial 3D reconstruction on each initial video frame within the range of the first to the second keyframe. The displacement reflects the motion trajectory of the matching feature points. That is, when the number of matching feature points and the length of the motion trajectory reach a certain condition, the initial reconstruction begins. By adding temporal constraints, the continuous and stable poses and spatial 3D points of some frames can be quickly recovered and the 3D scene can be reconstructed during the initial reconstruction process, effectively reducing the waiting time of objects on mobile devices and achieving faster interaction. Furthermore, after the initial reconstruction is completed, several incremental reconstructions are iteratively executed. For other video frames after the second keyframe, keyframes are also adaptively selected based on the number and displacement of matching feature points, thereby performing 3D reconstruction on each incremental video frame within the range of the second to the third keyframe, gradually optimizing the reconstruction results, improving reconstruction accuracy, and enhancing the final effect of AR effects.

[0657] In summary, this application, through feature matching and multi-threaded incremental optimization, can respond promptly to videos awaiting processing, effectively reducing object waiting time on mobile devices and achieving more accurate and faster interaction. Furthermore, this application proposes a fast spatial multi-plane detection algorithm to quickly support interactions such as object placement in spatial planes.

[0658] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0659] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0660] Having introduced the video-based 3D reconstruction method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0661] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0662] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The server 120 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 19 As shown, it includes a memory 1901, a communication module 1903, and one or more processors 1902.

[0663] The memory 1901 is used to store computer programs executed by the processor 1902. The memory 1901 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0664] Memory 1901 may be volatile memory, such as random-access memory (RAM); memory 1901 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1901 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1901 may be a combination of the above-described memories.

[0665] Processor 1902 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1902 is used to implement the above-described video-based 3D reconstruction method when it calls the computer program stored in memory 1901.

[0666] The communication module 1903 is used to communicate with terminal devices and other servers.

[0667] This application embodiment does not limit the specific connection medium between the memory 1901, communication module 1903, and processor 1902. This application embodiment... Figure 19 The memory 1901 and the processor 1902 are connected via a bus 1904, which is in the middle of the memory. Figure 19 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1904 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 19 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0668] The memory 1901 stores a computer storage medium containing computer-executable instructions for implementing the video-based 3D reconstruction method of this application embodiment. The processor 1902 executes the aforementioned video-based 3D reconstruction method, such as... Figure 4 As shown.

[0669] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the electronic device can be structured as follows: Figure 20As shown, it includes components such as: communication component 2010, memory 2020, display unit 2030, camera 2040, sensor 2050, audio circuit 2060, Bluetooth module 2070, and processor 2080.

[0670] The communication component 2010 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0671] The memory 2020 can be used to store software programs and data. The processor 2080 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 2020. The memory 2020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 2020 stores an operating system that enables the terminal device 110 to run. In this application, the memory 2020 may store the operating system and various applications, and may also store a computer program that executes the video-based 3D reconstruction method of the embodiments of this application.

[0672] The display unit 2030 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 110, in a graphical user interface (GUI). Specifically, the display unit 2030 may include a display screen 2032 disposed on the front of the terminal device 110. The display screen 2032 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 2030 can be used to display input videos, 3D reconstruction results, AR text effects, etc., as described in the embodiments of this application.

[0673] The display unit 2030 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 110. Specifically, the display unit 2030 may include a touch screen 2031 disposed on the front of the terminal device 110, which can collect touch operations of the user on or near it, such as clicking buttons, dragging scroll boxes, etc.

[0674] The touchscreen 2031 can be placed on top of the display screen 2032, or the touchscreen 2031 and the display screen 2032 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 2030 can display the application and the corresponding operation steps.

[0675] Camera 2040 can be used to capture still images, which users can then share via an application. There can be one or multiple cameras 2040. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 2080 for conversion into a digital image signal.

[0676] The terminal device may also include at least one sensor 2050, such as an accelerometer 2051, a proximity sensor 2052, a fingerprint sensor 2053, and a temperature sensor 2054. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0677] Audio circuitry 2060, speaker 2061, and microphone 2062 provide an audio interface between the user and terminal device 110. Audio circuitry 2060 converts received audio data into electrical signals, which are then transmitted to speaker 2061, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. Conversely, microphone 2062 converts collected sound signals into electrical signals, which are then received by audio circuitry 2060, converted back into audio data, and output to communication component 2010 for transmission to, for example, another terminal device 110, or to memory 2020 for further processing.

[0678] The Bluetooth module 2070 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 2070, thereby exchanging data.

[0679] The processor 2080 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 2020 and calling data stored in the memory 2020. In some embodiments, the processor 2080 may include one or more processing units; the processor 2080 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 2080. In this application, the processor 2080 can run an operating system, applications, user interface display and touch response, as well as the video-based 3D reconstruction method of this embodiment. Furthermore, the processor 2080 is coupled to the display unit 2030.

[0680] In some possible implementations, various aspects of the video-based 3D reconstruction method provided in this application can also be implemented as a program product, comprising a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the video-based 3D reconstruction method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 4 The steps are shown in the figure.

[0681] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0682] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0683] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0684] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0685] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0686] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0687] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0688] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0689] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0690] These computer program commands may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the commands stored in the computer-readable storage medium produce an article of manufacture including command means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0691] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0692] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0693] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A video-based 3D reconstruction method, characterized in that, The method includes: For the video to be processed, each matching feature point between frames and the position information of each matching feature point on the corresponding video frame are detected frame by frame; during the frame-by-frame detection process, the following reconstruction operation is performed based on the detected video frames, and the undetected video frames are further processed: The initial reconstruction operation is performed in the following manner: based on the number of each matching feature point and the inter-frame displacement, a first keyframe and a second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame. After the initial reconstruction operation is completed, incremental reconstruction operations are iteratively performed in the following manner until the video processing is completed: based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed on each incremental video frame; wherein, in subsequent iterations other than the first round of incremental reconstruction operations, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

2. The method as described in claim 1, characterized in that, The step-by-step detection of matching feature points between frames and the position information of each matching feature point on the corresponding video frame includes: For each video frame in the video, perform the following operations in sequence: If the number of feature points in the preceding video frame adjacent to the video frame is greater than a first number threshold, then optical flow tracing is used to determine each matching feature point between the video frame and the preceding video frame, as well as the position information of each matching feature point on the video frame and the preceding video frame. If the number of feature points in the preceding video frame adjacent to the video frame is not greater than a first number threshold, then each feature point on the video frame is extracted, and each matching feature point between the video frame and the preceding video frame is determined by optical flow tracing, as well as the position information of each matching feature point on the video frame and the preceding video frame.

3. The method as described in claim 2, characterized in that, The extraction of feature points on the video frame includes: At least one instruction set acceleration method is used to perform accelerated segment test feature extraction on the video frame to obtain each feature point contained in the video frame and the response value of each feature point; The video frame is divided into multiple filtering grids; For each filter grid, the feature points in the filter grid are filtered according to the response value of each feature point in the filter grid.

4. The method as described in claim 3, characterized in that, After filtering the feature points in each filter grid according to the response value of each feature point in the filter grid, the method further includes: If the number of feature points in the video frame is greater than the second number threshold, then at least one feature point with line features is deleted based on the difference between the gradient direction angles of each feature point in the video frame. For each feature point in the video frame, whether the feature point has line features is determined in the following way: According to the preset range parameters, the detection area is determined with the feature point as the center point; The gradient direction angle of each feature point in the region to be detected is determined, as well as the gradient interval to which each gradient direction angle belongs; wherein each gradient interval is obtained by dividing the range of gradient direction angle values ​​at equal intervals; Determine the target gradient interval containing the most feature points in each gradient interval, and the reference gradient interval containing the second most feature points. If the target gradient interval contains the center point, and the angle difference between the center point and the reference gradient interval exceeds a preset angle threshold, then the center point is determined to have line characteristics.

5. The method according to any one of claims 1 to 4, characterized in that, When detecting each matching feature point between frames and the position information of each matching feature point on the corresponding video frame, the method further includes: For each pair of adjacent frames in the video, perform at least one of the following verification operations: By performing random sampling consistency verification on each matching feature point in two adjacent frames, matching feature points that fail the verification are deleted. The preceding video frame in two adjacent frames is divided into multiple verification grids. For each matching feature point in each verification grid: if the difference between the optical flow displacement of the matching feature point in the two adjacent frames and the corresponding average optical flow displacement is greater than a preset difference threshold, the matching feature point is deleted. The average optical flow displacement is the average value of the optical flow displacement of each matching feature point in the verification grid to which the matching feature point belongs in the two adjacent frames.

6. The method according to any one of claims 1 to 4, characterized in that, The step of selecting a first keyframe and a second keyframe from the video based on the number of each matching feature point and the inter-frame displacement includes: The first frame of the video is taken as the initial i-th frame, and the second frame of the video is taken as the initial j-th frame; For the current i-th frame and j-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold. If the selection conditions are not met, let j = j + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame; If the selection conditions are met, pose estimation is performed on the current i-th frame and j-th frame, and based on the pose estimation results, the latest determined i-th frame is taken as the first keyframe, and the latest determined j-th frame is taken as the second keyframe.

7. The method as described in claim 6, characterized in that, The step of using the most recently determined i-th frame as the first keyframe and the most recently determined j-th frame as the second keyframe based on the pose estimation result includes: For the current i-th and j-th frames, determine the matching inliers between frames based on the pose estimation results; If the total number of all matched points is greater than the fourth number threshold, then the current i-th frame is taken as the first keyframe and the current j-th frame is taken as the second keyframe. If the total number of the matched points is not greater than the fourth quantity threshold and the total number of the matched feature points is not greater than the fifth quantity threshold, then let i = i + 1, and return to the step of analyzing whether the number of the matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame. If the total number of all matched intrapoints is not greater than the fourth quantity threshold, and the total number of all matched feature points is greater than the fifth quantity threshold, then the preset displacement threshold is increased, j = j + 1, and the process of analyzing whether the number of all matched feature points and the inter-frame displacement meet the selection conditions for the current i-th frame and j-th frame is returned.

8. The method according to any one of claims 1 to 4, characterized in that, The step of selecting a third keyframe after the second keyframe based on the number of matching feature points and the inter-frame displacement includes: Take the current second keyframe as the j-th frame, and take the first frame after the second keyframe as the initial k-th frame; For the j-th frame and the current k-th frame, analyze whether the number of each matching feature point and the inter-frame displacement meet the selection conditions; wherein, the selection conditions are: the number of matching feature points is greater than a third quantity threshold, and the average optical flow displacement of the matching feature points is greater than a preset displacement threshold. If the selection conditions are not met, let k = k + 1, and return to the step of analyzing whether the number of each matching feature point and the inter-frame displacement meet the selection conditions for the j-th frame and the current k-th frame; If the selection criteria are met, the newly determined k-th frame will be used as the third keyframe.

9. The method according to any one of claims 1 to 4, characterized in that, The pose information corresponding to each initial video frame within the range from the first keyframe to the second keyframe is determined in the following manner: Based on the camera intrinsic parameters and the position information of each matching feature point between the first keyframe and the second keyframe, the relative pose between the first keyframe and the second keyframe is determined. Based on the relative pose, determine the pose information corresponding to the first keyframe and the second keyframe when the first keyframe is used as the reference coordinate system; Based on the pose information corresponding to the first keyframe and the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to other initial video frames between the first keyframe and the second keyframe is determined frame by frame. Furthermore, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined in the following manner: Based on the pose information corresponding to the second keyframe, combined with the uniform motion model and the projection constraints of three-dimensional points and two-dimensional points, the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe is determined frame by frame.

10. The method as described in claim 1, characterized in that, The step of performing 3D reconstruction on each initial video frame based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, includes: Based on the pose information corresponding to the first keyframe and the second keyframe, and combined with the position information of each matching feature point between the first keyframe and the second keyframe, two-frame triangulation processing is performed on each matching feature point between the first keyframe and the second keyframe. For other initial video frames between the first keyframe and the second keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other initial video frames and their previous video frames, and combined with the position information of each matching feature point between the other initial video frames and their previous video frames, two-frame triangulation processing is performed on each matching feature point between the other initial video frames and their previous video frames. The 3D point cloud obtained through two frames of triangulation during the initial reconstruction operation will be used as the initial 3D reconstruction result.

11. The method as described in claim 10, characterized in that, The method further includes: Perform at least one of the following optimization operations on the initial 3D reconstruction results: Based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, and combined with the position information of each matching feature point between each initial video frame, multi-frame triangulation processing is performed on each matching feature point between each initial video frame; the three-dimensional point cloud obtained through multi-frame triangulation processing is added to the initial three-dimensional reconstruction result to optimize the initial three-dimensional reconstruction result. Global bundle adjustment (BA) is performed on the current initial 3D reconstruction results.

12. The method as described in claim 10, characterized in that, If the first keyframe is not the first frame in the video, then during the initial reconstruction operation, the method further includes: For the other video frames before the first keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the other video frames and the next video frame, and combined with the position information of each matching feature point between the other video frames and the next video frame, two-frame triangulation processing is performed on each matching feature point between the other video frames and the next video frame. The 3D point cloud obtained through two frames of triangulation is added to the current initial 3D reconstruction result to optimize the initial 3D reconstruction result.

13. The method as described in claim 1, characterized in that, The step of performing 3D reconstruction on each incremental video frame based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, combined with the position information, includes: For each incremental video frame within the range from the second keyframe to the third keyframe, after determining the corresponding pose information frame by frame, based on the pose information corresponding to the incremental video frame and its previous video frame, and combined with the position information of each matching feature point between the incremental video frame and its previous video frame, two-frame triangulation processing is performed on each matching feature point between the incremental video frame and its previous video frame. Based on the pose information corresponding to each incremental video frame within the range of the second keyframe to the third keyframe, and combined with the position information of each matching feature point between each incremental video frame, multi-frame triangulation is performed on each matching feature point between each incremental video frame. The 3D point cloud obtained through two-frame triangulation and multi-frame triangulation during this round of incremental reconstruction will be used as the result of this round of incremental 3D reconstruction. Incremental BA optimization was performed on the incremental 3D reconstruction results of this round.

14. The method according to any one of claims 10 to 12, characterized in that, After obtaining the 3D reconstruction result containing the 3D point cloud, the method further includes: The following plane detection operation is performed iteratively until a preset number of iterations is reached, or no plane is detected in the latest round of plane detection operation: From the 3D point cloud, select multiple 3D points that are currently not assigned to a plane. For each 3D point, perform the following operations: fit a candidate plane based on a preset number of nearest neighbors of the 3D point in the 3D point cloud; if the number of points in the candidate plane is higher than a preset threshold for the number of points in the plane, determine the average distance between the points in the plane and the candidate plane; wherein, the points in the plane are the 3D points among the currently unassigned 3D points that are less than a preset distance threshold from the candidate plane. Compare the various average distances determined in this round of planar detection operations to determine the minimum average distance; If the determined minimum average distance is not less than the preset error threshold, then it is determined that the plane cannot be detected in this round of plane detection operation; If the determined minimum average distance is less than the preset error threshold, the target plane is obtained by cleaning up the in-plane points of the target candidate plane corresponding to the minimum average distance.

15. The method as described in claim 14, characterized in that, The step of cleaning up in-plane points on the candidate target plane corresponding to the minimum average distance to obtain the target plane includes: Project the points in each plane corresponding to the target candidate plane onto the target candidate plane to obtain the corresponding two-dimensional points; Based on the distance between two-dimensional points, the target candidate plane is divided into two sub-planes; Determine the average distance between a two-dimensional point in each subplane and the center of the subplane, and the center distance between the centers of two subplanes; If the center distance is less than the target distance threshold, then the target candidate plane is determined as the target plane; the target distance threshold is positively correlated with the average distance corresponding to the larger of the two sub-planes; If the center distance is not less than the target distance threshold, then the larger plane is determined to be the target plane.

16. The method as described in claim 14, characterized in that, When performing incremental reconstruction, after iteratively performing plane detection operations, the method further includes: For each target plane that has been obtained, the target planes that meet the preset plane merging conditions are merged based on the normal vector, outer contour polygon, and outer contour polygon after mutual projection of each pair of target planes.

17. A video-based three-dimensional reconstruction device, characterized in that, include: The feature processing unit is used to detect, frame by frame, each matching feature point between frames and the position information of each matching feature point on the corresponding video frame for the video to be processed. The reconstruction unit is configured to perform the following reconstruction operation based on the detected video frames during the frame-by-frame detection process of the feature processing unit, and the feature processing unit is configured to continue processing undetected video frames: The initial reconstruction operation is performed in the following manner: based on the number of each matching feature point and the inter-frame displacement, a first keyframe and a second keyframe are selected from the video, and based on the pose information corresponding to each initial video frame within the range of the first keyframe to the second keyframe, combined with the position information, three-dimensional reconstruction is performed on each initial video frame. After the initial reconstruction operation is completed, incremental reconstruction operations are iteratively performed in the following manner until the video processing is completed: based on the number of each matching feature point and the inter-frame displacement, a third keyframe is selected after the second keyframe, and based on the pose information corresponding to each incremental video frame within the range from the second keyframe to the third keyframe, combined with the position information, three-dimensional reconstruction is performed on each incremental video frame; wherein, in subsequent iterations other than the first round of incremental reconstruction operations, the second keyframe in each round is: the third keyframe selected in the adjacent previous round.

18. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 16.

19. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 16.

20. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 16.

Images