Floating target three-dimensional reconstruction method based on deep learning

Abstract

The invention discloses a floating target three-dimensional reconstruction method based on deep learning, and relates to the technical field of three-dimensional reconstruction, a SuperPoint model is combined with RAFT and LO-RANSAC algorithms, the matching precision of feature points is remarkably improved, the RAFT model predicts the positions of the feature points through optical flow, the consistency of the matching of the feature points in adjacent frames in continuous motion is higher, and the matching precision of the feature points is improved. LO-RANSAC preferentially retains feature points with high confidence and stable positions through dynamic threshold adjustment and local optimization, eliminates points with low confidence and reduces the mismatching rate, meanwhile, an RAFT model carries out optical flow prediction on each feature point, the positions of the feature points are updated in real time, the track coherence of the feature points between adjacent frames is effectively kept, and the accuracy of the optical flow prediction is improved. According to the method, the drift and matching failure probability of feature points in fast motion is reduced, epipolar consistency constraint of a basic matrix and LO-RANSAC error elimination are combined, the precision of three-dimensional reconstruction is remarkably improved, and the accuracy and stability of three-dimensional reconstruction can still be kept in a motion scene.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a floating target three-dimensional reconstruction method based on deep learning. Background Art

[0002] Currently, most 3D reconstruction schemes have poor effects on floating targets, that is, targets in a moving environment. Floating targets have dynamic characteristics, such as irregular non-rigid motion, rapid rotation, and large displacement, which makes the feature points between frames in the image sequence unstable, resulting in an increase in the mismatch rate and reduced 3D reconstruction accuracy. In this context, how to extract stable and effective feature points from each frame of the image and track them continuously has become a key problem in the 3D reconstruction of floating targets.

[0003] Traditional 3D reconstruction methods mostly rely on classic feature point detection algorithms, such as SIFT and ORB. These algorithms generate feature points by capturing local texture or edge information on the surface of an object, and match these feature points between different frames in the image sequence to track the target's motion trajectory. These methods work well in scenes with stationary or slowly moving targets. However, in scenes where floating targets move rapidly or undergo non-rigid deformations, the appearance and position of feature points will change significantly in adjacent frames, making it difficult for traditional feature point detection and matching processes to remain stable. The drift, disappearance or excessive changes of feature points lead to unstable matching results, which in turn affects the reconstruction accuracy. In addition, methods such as SIFT and ORB lack the ability to resist background and occlusion changes, and are prone to extracting invalid features on background noise or occlusions, further exacerbating the risk of mismatching.

[0004] To address these challenges, some methods add a preliminary screening mechanism to the feature point screening and tracking process. For example, some mismatching points are eliminated through the random sampling consensus algorithm RANSAC based on geometric constraints, and the basic matrix is ​​solved on this basis. However, due to the randomness of the RANSAC sampling process, the mismatch elimination effect is not stable, and it is still easy to generate cumulative errors when the feature point mismatch rate is high. At the same time, the iterative calculation of RANSAC is often inefficient under the real-time reconstruction requirements of floating targets, affecting the speed and real-time performance of the reconstruction.

[0005] In addition, traditional solutions lack the ability to dynamically evaluate the stability of feature points. When floating targets move quickly, it is difficult to adjust detection and tracking parameters in time, resulting in the inability to distinguish between effective features and noise features when the scene changes dramatically. When floating targets move violently, it is very easy to mismatch, and it is difficult to ensure the accuracy and stability of three-dimensional reconstruction. Therefore, a floating target three-dimensional reconstruction method based on deep learning is urgently needed to solve such problems. Summary of the invention

[0006] In view of the above existing problems, the present invention is proposed.

[0007] The present invention provides a floating target three-dimensional reconstruction method based on deep learning to solve the problem that traditional solutions lack the ability to dynamically evaluate the stability of feature points. When the floating target moves quickly, it is difficult to adjust the detection and tracking parameters in time, resulting in the inability to distinguish between effective features and noise features when the scene changes drastically. When the floating target moves violently, it is very easy to mismatch, making it difficult to ensure the accuracy and stability of the three-dimensional reconstruction.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] The present invention provides a floating target three-dimensional reconstruction method based on deep learning, which comprises:

[0010] Step S1, feature point screening and stable extraction,

[0011] Perform feature point detection and use the SuperPoint model to process each frame of the image in the feature point detection stage to screen out high-stability feature points;

[0012] Step S2: Dynamic feature point tracking and updating.

[0013] Based on the high-stability feature points screened in step S1, the RAFT model is used for continuous dynamic tracking; a set of feature points that have been dynamically tracked and have high confidence is output;

[0014] Step S3, three-dimensional structure reconstruction and error correction,

[0015] The high-confidence feature point information outputted in step S2 is applied to three-dimensional structure reconstruction.

[0016] Furthermore, the SuperPoint model has strong anti-interference ability and can cope with the irregular movement of floating targets. At the same time, in order to enhance the reliability of feature points in non-rigid target motion scenes, in step S1, after screening out high-stability feature points, the extracted feature points are further screened to retain feature points with high confidence and strong repeatability in the local environment, so as to avoid low-confidence feature points affecting subsequent matching.

[0017] Furthermore, in step S1, the SuperPoint model is used to detect feature points, and the extracted feature points are screened according to their stability. Specifically:

[0018] In each frame image I, the SuperPoint model generates the feature point position and its corresponding descriptor. Assuming the size of the input image I is H×W, the SuperPoint model extracts a set of feature points through a convolutional neural network:

[0019] P={(x i ,y i ,d i )|i=1,2,…,N}, where P is the set of extracted feature points, (x i ,y i ) represents the position of the i-th feature point in the image, d i is the descriptor vector corresponding to the position of the i-th feature point, with a dimension of D. SuperPoint extracts feature points through local information of the image and generates a high-dimensional descriptor d i , improve the anti-interference ability of feature points;

[0020] Introduce a confidence screening mechanism into the detected feature point set P: Let the confidence of each feature point be c i , the SuperPoint model generates a confidence score according to the response strength of each feature point, sets the confidence threshold to τ, and selects a set of feature points with higher confidence: P′={(x i ,y i ,d i )|c i >τ}, where P′ is the set of feature points after confidence screening, c i represents the confidence of the i-th feature point, and τ is the preset confidence threshold;

[0021] Perform local environment screening on the feature points after confidence screening, assuming that the relative position difference of the feature points in adjacent frames is Δ i,j satisfy: If Δ i,j <∈, it is considered that feature points i and j have high repeatability in the local environment and are retained, otherwise feature points with low confidence are removed, where: Δ i,j represents the distance between the i-th and j-th feature points in space, ∈ is the maximum distance difference allowed in the local environment; after the above screening, the final high-stability feature point set P″ is obtained.

[0022] Furthermore, in step S2, the RAFT model estimates the position of each feature point in adjacent frames, continuously tracks the floating target, and maintains the continuity of the feature point trajectory.

[0023] Furthermore, in step S2, a variant of the random sampling consensus algorithm, the local optimized random sampling consensus algorithm LO-RANSAC, is used to dynamically adjust the matching threshold according to the confidence of the feature points, and preferentially matches the feature points with stable positions in the local range, and eliminates the features with low confidence.

[0024] Furthermore, in step S2, based on the high stability feature point set P″ screened in step S1, the RAFT model is used to continuously track the feature points in adjacent frames, and the LORANSAC algorithm is used to dynamically adjust the matching threshold:

[0025] Suppose the feature point set of the t-th frame image is The RAFT model uses the optical flow field F t Predict the position of feature points in the t+1th frame: in, is the position of the i-th feature point in the t-th frame, is the horizontal and vertical displacement of the corresponding feature point in the optical flow field on the tth frame, is the predicted position of the i-th feature point in the t+1-th frame; the RAFT model iteratively calculates the optical flow field to make the predicted position Maintain trajectory continuity in adjacent frames;

[0026] In the matching of each feature point, a confidence weight w is assigned to each feature point i :

[0027] Among them, d i is the distance error of the same feature point in the tth frame and the t+1th frame, σ is a tuning parameter used to control the sensitivity to small errors, w i The closer it is to 1, the smaller the position change of the feature point is, and it is suitable for matching.

[0028] Furthermore, in step S2, the LORANSAC algorithm is used for dynamic matching and feature point elimination, specifically:

[0029] The LORANSAC algorithm uses the confidence weight w i Dynamically adjust the matching threshold δ i , prioritize feature points with high confidence for matching, and eliminate feature points with low confidence. Set the matching threshold as δ i ,but:

[0030] Among them, α is the proportional coefficient, which controls the overall scale of the matching threshold. On this basis, if the matching distance d i >δ i , then the feature point is regarded as a feature with low confidence and is removed, and finally a set of feature points with high confidence and stable position is obtained;

[0031] The set of feature points after LORANSAC screening and elimination is recorded as P″ t+1 , as the set of dynamically tracked and highly confident feature points in the t+1th frame: Where P″t+1 Contains all the dynamic thresholds δ i feature points.

[0032] Furthermore, in step S3, the three-dimensional structure reconstruction step includes:

[0033] Construct the basic matrix and remove mismatched feature points through the polar line consistency constraint;

[0034] In each round of iteration, LO-RANSAC is combined to remove outliers that deviate from the trajectory, and the updated basic matrix is ​​used for reconstruction and error correction.

[0035] Furthermore, in step S3, the high confidence feature point set P″ output in step S2 is t+1 Applied to three-dimensional structure reconstruction and construction of basic matrix:

[0036] In two adjacent frames, namely, frame t and frame t+1, assuming that P t and P t+1 is the set of matched feature points in these two frames. First, the basic matrix F is constructed, which satisfies the following relationship:

[0037] Among them, p t =(x t ,y t ,1) T represents the homogeneous coordinates of a feature point in the tth frame, p t+1 =(x t+1 ,y t+1 ,1) T represents the homogeneous coordinates of the corresponding feature points in the t+1th frame, F is a 3×3 basic matrix, describing the geometric constraints between two adjacent frames, through multiple feature point pairs (p t ,p t+1 ), and use the least squares method to solve the basic matrix F;

[0038] On the constructed basic matrix F, the polar line consistency constraint is applied to remove the mismatched feature points. For each feature point pair (p t ,p t+1 ), satisfying the following epipolar constraints: Among them, d(p t+1 ,Fp t ) represents the feature point p t+1 The distance to the pole, (Fp t ) x and (Fp t ) y Fp t The horizontal and vertical components of ∈ are error thresholds, which are used to constrain whether the feature points are near the epipolar line. If the epipolar line distance d(p t+1,Fp t ) exceeds the threshold ∈, the point is considered to be a mismatched point and is removed.

[0039] Furthermore, in step S3, after initially eliminating the mismatched points, the remaining feature point set P is quantized by LORANSAC. inliers Further screening, for each feature point, define the error measure δ of deviation from the trajectory i :

[0040] Among them, p t+1,i is the observed position of the i-th feature point in the t+1-th frame, is the ideal feature point position calculated based on the basic matrix F, δ i Indicates the error between the observed point and the ideal position;

[0041] In LORANSAC, the matching threshold of each feature point is dynamically adjusted to τ according to its confidence. i : Among them, α is the proportional coefficient, which is used to adjust the overall matching threshold scale, w i is the confidence weight of the i-th feature point. The higher the confidence, the stricter the matching requirement. If the deviation error δ of a feature point i Exceeding the dynamic threshold τ i , then it will be removed, and the final retained point set is high confidence and consistent with the trajectory;

[0042] For the feature point set after the epipolar consistency and LORANSAC screening, the basic matrix is ​​used for three-dimensional reconstruction, and the three-dimensional coordinate point X is obtained by triangulation. i =(X i ,Y i ,Z i ):

[0043] Among them, X i Represents the three-dimensional coordinates of the feature point, π j (X) is the camera projection function, which projects the 3D point X onto the image plane of the jth frame. j represents the index of the adjacent frame, which is taken as frames t and t+1. That is, the 3D point reconstruction is performed between the tth frame and the t+1th frame. j,i It represents the pixel coordinates of the i-th feature point observed on the j-th frame image plane. In each iteration, the three-dimensional position of the feature point is further optimized by updating the basic matrix F and the reconstruction error constraint, thereby ensuring the stability and accuracy of the three-dimensional reconstruction.

[0044] The beneficial effects of the present invention are:

[0045] The present invention adopts the SuperPoint model to improve the stability of feature point extraction. In fast-motion scenes, it can extract feature points with stronger anti-interference ability. At the same time, it combines RAFT and LO-RANSAC to significantly improve the accuracy of feature point matching. The RAFT model predicts the position of feature points through optical flow, so that the feature point matching in adjacent frames of continuous motion has higher coherence. LO-RANSAC uses dynamic threshold adjustment and local optimization to give priority to retaining feature points with high confidence and stable position, eliminate low-confidence points, and reduce the mismatch rate.

[0046] In the present invention, the RAFT model performs optical flow prediction on each feature point, updates the position of the feature point in real time, effectively maintains the trajectory continuity of the feature point between adjacent frames, and reduces the drift of the feature point in fast motion and the probability of matching failure.

[0047] The present invention combines the polar line consistency constraint of the basic matrix with the LO-RANSAC error elimination, which significantly improves the accuracy of three-dimensional reconstruction and can still maintain the accuracy and stability of three-dimensional reconstruction in complex motion scenes. In addition, when the triangulation method is used to solve the three-dimensional coordinates, the reconstruction accuracy is further optimized through iterative updating of the basic matrix and error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0049] Figure 1 It is a schematic flow chart of the floating target three-dimensional reconstruction method based on deep learning of the present invention. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1, reference Figure 1 This embodiment provides a floating target three-dimensional reconstruction method based on deep learning, comprising the following steps:

[0054] Step S1, feature point screening and stable extraction,

[0055] Perform feature point detection and use the SuperPoint model to process each frame of the image in the feature point detection stage to screen out high-stability feature points;

[0056] The SuperPoint model has strong anti-interference ability and can cope with the irregular movement of floating targets. At the same time, in order to enhance the reliability of feature points in non-rigid target motion scenes, in step S1, after screening out high-stability feature points, the extracted feature points are further screened to retain feature points with high confidence and strong repeatability in the local environment, so as to avoid low-confidence feature points affecting subsequent matching.

[0057] In step S1, the SuperPoint model is used to detect feature points, and the extracted feature points are screened according to their stability. Specifically:

[0058] In each frame image I, the SuperPoint model generates the feature point position and its corresponding descriptor. Assuming the size of the input image I is H×W, the SuperPoint model extracts a set of feature points through a convolutional neural network:

[0059] P={(x i ,y i ,d i )|i=1,2,…,N}, where P is the set of extracted feature points, (x i ,y i ) represents the position of the i-th feature point in the image, d i is the descriptor vector corresponding to the position of the i-th feature point, with a dimension of D. SuperPoint extracts feature points through local information of the image and generates a high-dimensional descriptor d i , improve the anti-interference ability of feature points;

[0060] Introduce a confidence screening mechanism into the detected feature point set P: Let the confidence of each feature point be c i, the SuperPoint model generates a confidence score according to the response strength of each feature point, sets the confidence threshold to τ, and selects a set of feature points with higher confidence: P′={(x i ,y i ,d i )|c i >τ}, where P′ is the set of feature points after confidence screening, c i represents the confidence of the i-th feature point, and τ is the preset confidence threshold;

[0061] Perform local environment screening on the feature points after confidence screening, assuming that the relative position difference of the feature points in adjacent frames is Δ i,j satisfy: If Δ i,j <∈, it is considered that feature points i and j have high repeatability in the local environment and are retained, otherwise feature points with low confidence are removed, where: Δ i,j represents the distance between the i-th and j-th feature points in space, ∈ is the maximum distance difference allowed in the local environment; after the above screening, the final high-stability feature point set P″ is obtained;

[0062] Specifically, this method uses the SuperPoint model in feature point extraction to enhance the anti-interference ability under irregular motion. The SuperPoint model can efficiently extract feature points and provide higher stability in irregular motion and non-rigid deformation. It eliminates noise features and low-confidence points through confidence screening mechanism and local environment screening.

[0063] Step S2: Dynamic feature point tracking and updating.

[0064] Based on the high-stability feature points screened in step S1, the RAFT model is used for continuous dynamic tracking; a set of feature points that have been dynamically tracked and have high confidence is output;

[0065] In step S2, the RAFT model estimates the position of each feature point in adjacent frames, continuously tracks the floating target, and maintains the continuity of the feature point trajectory;

[0066] In step S2, a variant of the random sampling consensus algorithm, the local optimized random sampling consensus algorithm LO-RANSAC, is used to dynamically adjust the matching threshold according to the confidence of the feature points, and preferentially matches the feature points with stable positions in the local range, and eliminates the features with low confidence.

[0067] In step S2, based on the high stability feature point set P″ screened in step S1, the RAFT model is used to continuously track the feature points in adjacent frames, and the LORANSAC algorithm is used to dynamically adjust the matching threshold:

[0068] Suppose the feature point set of the t-th frame image is The RAFT model uses the optical flow field F t Predict the position of feature points in the t+1th frame: in, is the position of the i-th feature point in the t-th frame, is the horizontal and vertical displacement of the corresponding feature point in the optical flow field on the tth frame, is the predicted position of the i-th feature point in the t+1-th frame; the RAFT model iteratively calculates the optical flow field to make the predicted position Maintain trajectory continuity in adjacent frames;

[0069] In the matching of each feature point, a confidence weight w is assigned to each feature point i :

[0070] Among them, d i is the distance error of the same feature point in the tth frame and the t+1th frame, σ is a tuning parameter used to control the sensitivity to small errors, w i The closer it is to 1, the smaller the position change of the feature point is, and it is suitable for matching;

[0071] In step S2, the LORANSAC algorithm is used for dynamic matching and feature point elimination. Specifically:

[0072] The LORANSAC algorithm uses the confidence weight w i Dynamically adjust the matching threshold δ i , prioritize feature points with high confidence for matching, and eliminate feature points with low confidence. Set the matching threshold as δ i ,but:

[0073] Among them, α is the proportional coefficient, which controls the overall scale of the matching threshold. On this basis, if the matching distance d i >δ i , then the feature point is regarded as a feature with low confidence and is removed, and finally a set of feature points with high confidence and stable position is obtained;

[0074] The set of feature points after LORANSAC screening and elimination is recorded as P″ t+1 , as the set of dynamically tracked and highly confident feature points in the t+1th frame: Where P″ t+1 Contains all the dynamic thresholds δ i The characteristic points of

[0075] Specifically, in the feature point tracking stage, the RAFT model is used for continuous dynamic tracking, and the LO-RANSAC algorithm is combined to dynamically adjust the feature point matching. The RAFT model effectively copes with the rotation, displacement and deformation changes of floating targets through optical flow prediction, so that the feature points maintain trajectory continuity between adjacent frames. LO-RANSAC further dynamically adjusts the matching threshold according to the confidence of the feature points, eliminates mismatched points and low-confidence points, and ensures the accuracy and stability of the matching; in addition, LO-RANSAC is combined with the dynamic adjustment mechanism of confidence, which can give priority to matching feature points with high confidence in local areas, and has better anti-interference ability against occlusion and perspective changes.

[0076] Step S3, three-dimensional structure reconstruction and error correction,

[0077] Applying the high confidence feature point information output from step S2 to three-dimensional structure reconstruction;

[0078] In step S3, the three-dimensional structure reconstruction step includes:

[0079] Construct the basic matrix and remove mismatched feature points through the polar line consistency constraint;

[0080] In each iteration, LO-RANSAC is combined to remove outliers that deviate from the trajectory, and the updated basic matrix is ​​used for reconstruction and error correction;

[0081] In step S3, the high confidence feature point set P" output in step S2 is t+1 Applied to three-dimensional structure reconstruction and construction of basic matrix:

[0082] In two adjacent frames, namely, frame t and frame t+1, assuming that P t and P t+1 is the set of matched feature points in these two frames. First, the basic matrix F is constructed, which satisfies the following relationship:

[0083] Among them, p t =(x t ,y t ,1) T represents the homogeneous coordinates of a feature point in the tth frame, p t+1 =(x t+1 ,y t+1 ,1) T represents the homogeneous coordinates of the corresponding feature points in the t+1th frame, F is a 3×3 basic matrix, describing the geometric constraints between two adjacent frames, through multiple feature point pairs (p t ,p t+1 ), and use the least squares method to solve the basic matrix F;

[0084] On the constructed basic matrix F, the polar line consistency constraint is applied to remove the mismatched feature points. For each feature point pair (p t ,p t+1 ), satisfying the following epipolar constraints: Among them, d(p t+1 ,Fp t ) represents the feature point p t+1 The distance to the pole, (Fp t ) x and (Fp t ) y Fp t The horizontal and vertical components of ∈ are error thresholds, which are used to constrain whether the feature points are near the epipolar line. If the epipolar line distance d(p t+1 ,Fp t ) exceeds the threshold ∈, the point is considered as a mismatch point and is removed;

[0085] In step S3, after initially eliminating the mismatched points, the remaining feature point set P is quantified by LORANSAC. inliers Further screening, for each feature point, define the error measure δ of deviation from the trajectory i :

[0086] Among them, p t+1,i is the observed position of the i-th feature point in the t+1-th frame, is the ideal feature point position calculated based on the basic matrix F, δ i Indicates the error between the observed point and the ideal position;

[0087] In LORANSAC, the matching threshold of each feature point is dynamically adjusted to τ according to its confidence. i : Among them, α is the proportional coefficient, which is used to adjust the overall matching threshold scale, w i is the confidence weight of the i-th feature point. The higher the confidence, the stricter the matching requirement. If the deviation error δ of a feature point i Exceeding the dynamic threshold τ i , then it will be removed, and the final retained point set is high confidence and consistent with the trajectory;

[0088] For the feature point set after the epipolar consistency and LORANSAC screening, the basic matrix is ​​used for three-dimensional reconstruction, and the three-dimensional coordinate point X is obtained by triangulation. i =(X i ,Y i ,Z i ):

[0089] Among them, X iRepresents the three-dimensional coordinates of the feature point, π j (X) is the camera projection function, which projects the 3D point X onto the image plane of the jth frame. j represents the index of the adjacent frame, which is taken as frames t and t+1. That is, the 3D point reconstruction is performed between the tth frame and the t+1th frame. j,i represents the pixel coordinates of the i-th feature point observed on the j-th frame image plane. In each iteration, the three-dimensional position of the feature point is further optimized by updating the basic matrix F and the reconstruction error constraint, thereby ensuring the stability and accuracy of the three-dimensional reconstruction;

[0090] Specifically, in the three-dimensional reconstruction stage, the epipolar consistency constraint of the basic matrix and the iterative optimization strategy of LO-RANSAC are combined to significantly reduce the impact of mismatching on the reconstruction accuracy. By constructing the basic matrix and utilizing the epipolar consistency to eliminate external points that deviate from the trajectory, only valid matching features are used for reconstruction, and the triangulation method is used to further improve the accuracy of the three-dimensional reconstruction. At the same time, dynamic optimization is achieved through the basic matrix and error correction process, which can maintain high accuracy and consistency even in complex motion conditions.

[0091] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A floating target three-dimensional reconstruction method based on deep learning, characterized by: include, Step S1, feature point screening and stable extraction, Perform feature point detection and use the SuperPoint model to process each frame of the image in the feature point detection stage to screen out high-stability feature points; Step S2: Dynamic feature point tracking and updating. Based on the high-stability feature points screened in step S1, the RAFT model is used for continuous dynamic tracking; a set of feature points that have been dynamically tracked and have high confidence is output; Step S3, three-dimensional structure reconstruction and error correction, The high-confidence feature point information outputted in step S2 is applied to three-dimensional structure reconstruction.

2. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 1, characterized in that: In step S1, after the feature points with high stability are screened out, the extracted feature points are further screened.

3. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 2, characterized in that: In step S1, the SuperPoint model is used to detect feature points, and the extracted feature points are screened according to their stability. Specifically: In each frame image I, the SuperPoint model generates the feature point position and its corresponding descriptor. Assuming the size of the input image I is H×W, the SuperPoint model extracts a set of feature points through a convolutional neural network: P = {(x i ,y i ,d i )|i=1,2,…,N}, where P is the set of extracted feature points, (x i ,y i ) represents the position of the i-th feature point in the image, d i is the descriptor vector corresponding to the position of the i-th feature point, with a dimension of D; Introduce a confidence screening mechanism into the detected feature point set P: Let the confidence of each feature point be c i , the SuperPoint model generates a confidence score according to the response strength of each feature point, sets the confidence threshold to τ, and selects a set of feature points with higher confidence: P′={(x i ,y i ,d i )|c i >τ}, where P′ is the set of feature points after confidence screening, c i represents the confidence of the i-th feature point, and τ is the preset confidence threshold; Perform local environment screening on the feature points after confidence screening, assuming that the relative position difference of the feature points in adjacent frames is Δ i,j satisfy: If Δ i,j <∈, it is considered that feature points i and j have high repeatability in the local environment and are retained, otherwise feature points with low confidence are removed, where: Δ i,j It represents the distance between the i-th and j-th feature points in space, and ∈ is the maximum distance difference allowed in the local environment.

4. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 3, characterized in that: In step S2, the RAFT model estimates the position of each feature point in adjacent frames and continuously tracks the floating target.

5. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 4, characterized in that: In step S2, the local optimized random sampling consensus algorithm LO-RANSAC is used to dynamically adjust the matching threshold according to the confidence of the feature points, and preferentially match the feature points with stable positions in the local range, and eliminate the features with low confidence.

6. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 5, characterized in that: In step S2, based on the high stability feature point set P″ screened in step S1, the RAFT model is used to continuously track the feature points in adjacent frames, and the LORANSAC algorithm is used to dynamically adjust the matching threshold: Suppose the feature point set of the t-th frame image is The RAFT model uses the optical flow field F t Predict the position of feature points in the t+1th frame: in, is the position of the i-th feature point in the t-th frame, is the horizontal and vertical displacement of the corresponding feature point in the optical flow field on the tth frame, is the predicted position of the i-th feature point in the t+1-th frame; In the matching of each feature point, a confidence weight w is assigned to each feature point i : Among them, d i is the distance error of the same feature point in the tth frame and the t+1th frame, σ is a tuning parameter used to control the sensitivity to small errors, w i The closer it is to 1, the smaller the position change of the feature point is, and it is suitable for matching.

7. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 6, characterized in that: In step S2, the LORANSAC algorithm is used for dynamic matching and feature point elimination. Specifically: The LORANSAC algorithm uses the confidence weight w i Dynamically adjust the matching threshold δ i , prioritize feature points with high confidence for matching, and eliminate feature points with low confidence. Set the matching threshold as δ i ,but: Among them, α is the proportional coefficient, which controls the overall scale of the matching threshold. On this basis, if the matching distance d i >δ i , then the feature point is regarded as a feature with low confidence and is removed; The set of feature points after LORANSAC screening and elimination is recorded as P″ t+1 , as the set of dynamically tracked and highly confident feature points in the t+1th frame: Where P″ t+1 Contains all the dynamic thresholds δ i feature points.

8. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 7, characterized in that: In step S3, the three-dimensional structure reconstruction step includes: Construct the basic matrix and remove mismatched feature points through the polar line consistency constraint; In each round of iteration, LO-RANSAC is combined to remove outliers that deviate from the trajectory, and the updated basic matrix is ​​used for reconstruction and error correction.

9. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 8, characterized in that: In step S3, the high confidence feature point set P" output in step S2 is t+1 Applied to three-dimensional structure reconstruction and construction of basic matrix: In two adjacent frames, namely, frame t and frame t+1, assuming that P t and P t+1 is the set of matched feature points in these two frames. First, the basic matrix F is constructed, which satisfies the following relationship: Among them, p t =(x t ,y t ,1) T represents the homogeneous coordinates of a feature point in the tth frame, p t+1 =(x t+1 ,y t+1 ,1) T represents the homogeneous coordinates of the corresponding feature points in the t+1th frame, and F is a 3×3 basic matrix that describes the geometric constraints between two adjacent frames; On the constructed basic matrix F, the polar line consistency constraint is applied to remove the mismatched feature points. For each feature point pair (p t ,p t+1 ), satisfying the following epipolar constraints: Among them, d(p t+1 ,Fp t ) represents the feature point p t+1 The distance to the epipolar line, (Fp t ) x and (Fp t ) y Fp t The horizontal and vertical components of ∈ are error thresholds, which are used to constrain whether the feature points are near the epipolar line. If the epipolar line distance d(p t+1 ,Fp t ) exceeds the threshold ∈, the point is considered to be a mismatched point and is removed.

10. The method for three-dimensional reconstruction of floating objects based on deep learning according to claim 9, characterized in that: In step S3, after initially eliminating the mismatched points, the remaining feature point set P is quantified by LORANSAC. inliers Further screening, for each feature point, define the error measure δ of deviation from the trajectory i : Among them, p t+1,i is the observed position of the i-th feature point in the t+1-th frame, is the ideal feature point position calculated based on the basic matrix F, δ i Represents the error between the observed point and the ideal position; In LORANSAC, the matching threshold of each feature point is dynamically adjusted to τ according to its confidence. i : Among them, α is the proportional coefficient, which is used to adjust the overall matching threshold scale, w i is the confidence weight of the i-th feature point. The higher the confidence, the stricter the matching requirement. If the deviation error δ of a feature point i Exceeding the dynamic threshold τ i , then remove it; For the feature point set after the epipolar consistency and LORANSAC screening, the basic matrix is ​​used for three-dimensional reconstruction, and the three-dimensional coordinate point X is obtained by triangulation. i =(X i ,Y i ,Z i ): Among them, X i Represents the three-dimensional coordinates of the feature point, π j (X) is the camera projection function, which projects the 3D point X onto the image plane of the jth frame. j represents the index of the adjacent frame, which is taken as frames t and t+1. That is, the 3D point reconstruction is performed between the tth frame and the t+1th frame. j,i Represents the pixel coordinates of the i-th feature point observed on the j-th frame image plane.

Images