Deep learning based space-based multi-view three-dimensional reconstruction method and system for space targets

Abstract

The application discloses a kind of based on deep learning's space-based space target multi-view three-dimensional reconstruction method and system, belong to three-dimensional reconstruction technical field, to solve the technical problem that space-based space target is difficult to high-precision three-dimensional reconstruction due to complex illumination, weak texture surface, size variation is big.It is first by saliency target detection network to the multi-view original image is removed background preprocessing, then by motion recovery structure processing obtains sparse three-dimensional point cloud and camera internal and external parameter, subsequently by multi-view depth estimation network predicts each perspective dense depth map, finally based on photometric and geometric consistency constraint completes depth map fusion, outputs space target three-dimensional point cloud reconstruction result.The reconstruction precision of the application is high, scene robustness is strong, can adapt to space-based on-orbit service lightweight deployment demand, and is widely applicable to space-based space target on-orbit three-dimensional reconstruction scene.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, specifically relating to a deep learning-based multi-view method for space-based targets. Figure Three Reconstruction methods and systems. Background Technology

[0002] With the rapid development of the space industry, the demands for missions such as space-based servicing, space situational awareness, and on-orbit maintenance and dismantling are constantly upgrading. Accurately acquiring the complete three-dimensional structure and high-precision morphology of space-based targets has become a core prerequisite for the successful execution of on-orbit servicing missions. Space-based detection platforms can acquire multi-view images of targets through flexible observation modes such as escort and fly-around. Compared with ground observation and lidar detection, three-dimensional reconstruction schemes based on passive stereo vision have become the mainstream technology direction for three-dimensional reconstruction of space-based targets due to their advantages of small size, light weight, and low power consumption.

[0003] Current stereo vision-based 3D reconstruction technologies are mainly divided into monocular, binocular, and multi-view methods. Figure Three The two approaches are: monocular 3D reconstruction, binocular 3D reconstruction, and binocular 3D reconstruction. Monocular 3D reconstruction lacks baseline constraints and cannot directly recover the true scale of the target. It requires the introduction of additional prior information or auxiliary sensors, which makes it difficult to meet the absolute scale reconstruction requirements of space-based on-orbit services. Binocular 3D reconstruction relies on high-precision camera calibration and adjustment, and requires a large baseline distance to achieve large depth range perception. The hardware deployment is difficult and costly, and it cannot meet the lightweight and miniaturized design requirements of space-based platforms.

[0004] Multiview Figure Three Multi-view stereo reconstruction technology, with its advantages in multi-view geometric constraints, has become the most promising solution for space-based scenarios. However, existing technologies still have many technical shortcomings that make them difficult to adapt to space-based environments: Traditional multi-view stereo matching algorithms, while possessing basic reconstruction accuracy, are prone to feature matching failures, incomplete depth information estimation, and missing reconstruction structures under conditions of drastic lighting changes and weak / no texture on target surfaces in space-based scenarios; Existing deep learning-based multi-view reconstruction schemes do not perform dedicated preprocessing for the complex backgrounds of space-based environments, and background interference such as stars and stray light can easily lead to inconsistent geometric transformations between the target and the background, directly reducing reconstruction accuracy; At the same time, the feature extraction structures of existing networks are insufficient in representing weakly textured regions, and depth prediction has poor robustness to complex lighting and target occlusion, and it is difficult to balance reconstruction accuracy and inference efficiency under the limited computing power of space-based platforms, failing to meet the engineering requirements of real-time on-orbit processing in space. All of the above technical problems directly limit multi-view... Figure Three The practical application of 3D reconstruction technology in space-based target observation scenarios is also a core technical pain point that this invention specifically addresses. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a deep learning-based multi-view system for space-based targets. Figure Three The method and system for 3D reconstruction are designed to address the technical challenges in existing technologies where space-based targets are difficult to reconstruct in 3D due to their complex lighting environments, weakly textured surfaces, and large size variations.

[0006] The present invention adopts the following technical solution: Deep Learning-Based Multi-View Target Application in Space Figure Three The 3D reconstruction method includes the following steps: S1. Obtain the original images of the same spatial target from multiple observation perspectives, input the original images into a salient target detection network to obtain a foreground mask, and remove the background from the original images based on the foreground mask to obtain a preprocessed multi-view image. S2. Perform motion reconstruction processing on the preprocessed multi-view image. The motion reconstruction processing includes feature point extraction, cross-view feature matching, geometric consistency verification, incremental image registration, triangulation and bundling adjustment to obtain sparse 3D point cloud and camera intrinsic and extrinsic parameters corresponding to each view. S3. Input the preprocessed multi-view image and its corresponding camera intrinsic and extrinsic parameters into the multi-view depth estimation network to perform dense depth prediction for each view and output dense depth maps for each view. S4. Based on the camera parameters of each view and the corresponding dense depth map, back projection is performed to generate candidate 3D points. Based on the photometric consistency constraint and geometric consistency constraint, the candidate 3D points are jointly screened and multi-view fused to output the 3D point cloud reconstruction result of the spatial target.

[0007] Preferably, in step S1, the salient object detection network is an InSPyReNet network, and the foreground mask generation and background removal processing specifically include: The original images from multiple observation perspectives are input into the pre-trained InSPyReNet network. The deep convolutional backbone network extracts multi-level features from the input images and constructs an image pyramid that covers low-level details to high-level semantic information. The Laplacian pyramid residuals of the corresponding saliency maps are predicted at multiple scales. Starting from the coarsest scale, the saliency maps reconstructed from the previous level are combined with the Laplacian residuals at the current scale to reconstruct a high-precision saliency map with the original resolution level by level. The high-precision saliency map is binarized to generate a binary foreground mask. The foreground mask is then multiplied pixel by pixel with the original input image to obtain a preprocessed multi-view image after removing the background.

[0008] Preferably, the motion recovery structure processing in step S2 specifically includes: S201. Extract feature points from the preprocessed multi-view images and perform feature matching between pairs of multi-view images to obtain overlapping areas of scenes and matching point pairs. S202. Using the fundamental matrix, essential matrix, or homography matrix, combined with the random sampling consensus algorithm, filter out outliers in feature matching and retain matching point pairs that meet geometric consistency. S203. Select two non-panoramic views to complete scene initialization. Based on the existing matching point pairs and scene model, complete the image registration of the remaining views through the PNP algorithm, and expand the scene's 3D point set through triangulation. S204. Construct a nonlinear least squares optimization problem using the bundle adjustment method. Simultaneously optimize the camera intrinsic parameters, camera extrinsic parameters, and 3D point coordinates of all views. Minimize the sum of squared errors of all 3D points reprojected onto the corresponding view imaging plane. Finally, output the optimized sparse 3D point cloud, camera intrinsic parameters, and camera extrinsic parameters of each view.

[0009] Preferably, the dense depth prediction in step S3 specifically includes the following steps: S301. Using iterative feature pyramid, dilated spatial convolutional pooling pyramid and variable convolutional network, multi-scale deep feature extraction is performed on the input preprocessed multi-view image to obtain multi-scale feature maps corresponding to each view. S302. Obtain the depth prediction range based on the camera intra-camera parameters and camera extrinsic parameters of each view. Map the feature maps of different viewpoints to the depth direction of the reference viewpoint through homography transformation on different depth planes. Construct a cost volume through view similarity calculation and perform regularization processing on the cost volume through three-dimensional convolution to obtain the depth probability volume. S303. Based on the confidence levels corresponding to different depths in the depth probability volume, perform a weighted average regression along the depth direction to obtain the dense depth map of the corresponding view.

[0010] Preferably, in step S301, the iterative feature pyramid includes a bottom-up encoder path and a top-down decoder path. The encoder path downsamples the initial features multiple times using a convolutional neural network to generate feature maps with decreasing scale and increasing channel number at various resolutions. The decoder path starts from the deepest small-scale features, expands the feature size through upsampling, and then adds and fuses the upsampled features with the same spatial scale in the encoder path element by element after channel adjustment, thereby realizing bidirectional iterative interaction between deep semantic information and shallow detail information. The dilated spatial convolutional pooling pyramid module sets up multiple dilated convolutional layers with different dilation rates in parallel, as well as a global average pooling branch. After concatenating the output features of all parallel branches in the channel dimension, it is fused and dimensionality reduced by 1×1 convolution to output a feature map with multi-scale contextual representation capabilities. The variable convolutional network learns two-dimensional offsets from the features of the previous layer through additional convolutional layers, and dynamically adjusts the spatial position of the convolutional sampling according to the two-dimensional offsets, so that the receptive field of the network adaptively focuses on the target edges and corner regions of the image.

[0011] Preferably, in step S302, the feature similarity between different views is calculated using a variance-based cost index to construct an initial cost body; the mapping relationship of the variance-based cost index is as follows:

[0012] in, The volume dimensions of the feature body; The number of views participating in the aggregation. This represents the number of effective pixels.

[0013] Preferably, in step S303, a weighted sum of all assumed depth values ​​is calculated along the depth direction to obtain the depth value of the corresponding pixel. The calculation formula is as follows:

[0014] in, For depth value The estimated probability value of the location. This represents the maximum value of the depth prediction range. This is the minimum value of the depth prediction range.

[0015] Preferably, during the training process of the multi-view depth estimation network, the loss function used is a weighted sum of L1 losses from the multi-scale depth prediction results. The expression is:

[0016] in, The number of views participating in the aggregation; The layer number of the network depth prediction structure; These are the weighting coefficients for each layer; This represents the number of layers in the network structure.

[0017] Preferably, step S4 specifically includes: S401. For the dense depth map of each view, the effective pixels with depth values ​​within a reasonable range and their depth values ​​are back-projected to the world coordinate system using camera geometry principles to generate candidate 3D point clouds for the corresponding view. S402, Perform multi-view analysis on candidate 3D point clouds Figure One A consistency check is performed by reprojecting candidate 3D points onto adjacent views. If a point satisfies both photometric consistency constraints and geometric consistency constraints, it is retained; otherwise, it is discarded. The photometric consistency constraint is that the photometric error of the candidate 3D point after reprojection is less than a preset threshold, and the geometric consistency constraint is that the error between the depth value of the candidate 3D point after reprojection and the depth value of that position in the dense depth map of the corresponding view is less than a preset threshold. S403. Output the filtered and fused 3D point cloud as the 3D reconstruction result of the space-based target.

[0018] Secondly, embodiments of the present invention provide a deep learning-based multi-view system for space-based targets. Figure Three 3D reconstruction system, including: The preprocessing module is used to acquire the original images of the same spatial target from multiple viewing angles, input the original images into the salient target detection network to obtain the foreground mask, and remove the background of the original images based on the foreground mask to output the preprocessed multi-view images. The recovery module is used to perform motion recovery structure processing on the preprocessed multi-view image and output sparse 3D point cloud as well as the camera intra-camera parameters and camera extrinsic parameters corresponding to each view. The prediction module is used to input the preprocessed multi-view image and its corresponding camera intrinsic and extrinsic parameters into the multi-view depth estimation network, perform dense depth prediction for each view, and output dense depth maps for each view. The reconstruction module is used to back-project the camera parameters of each view with the corresponding dense depth map to generate candidate 3D points, and to perform joint screening and multi-view fusion of the candidate 3D points based on photometric consistency constraints and geometric consistency constraints, and output the 3D point cloud reconstruction result of the spatial target.

[0019] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned deep learning-based multi-view of space-based targets. Figure Three The steps of the dimensional reconstruction method.

[0020] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the aforementioned deep learning-based multi-view space-based target detection. Figure Three The steps of the dimensional reconstruction method.

[0021] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned deep learning-based multi-view space-based target detection. Figure Three The steps of the dimensional reconstruction method.

[0022] Sixthly, embodiments of the present invention provide an electronic device including a computer program, which, when executed by the electronic device, implements the aforementioned deep learning-based multi-view of space-based targets. Figure Three The steps of the dimensional reconstruction method.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: A Deep Learning-Based Multi-View Space Target Approach Figure Three This reconstruction method first eliminates interference from the complex space background at the source through salient target detection, solving the reconstruction deviation problem caused by the inconsistency between the geometric transformation of the target and the background. It then recovers the structure through single-camera multi-view motion, addressing the industry pain points of monocular solutions failing to restore true scale and binocular solutions being cumbersome to calibrate and costly. Finally, it combines deep learning multi-view depth prediction with a dual consistency constraint fusion scheme to overcome the core challenges of weak texture and incomplete depth estimation under complex lighting conditions for space-based targets. This invention restores true scale, improves the completeness and accuracy of reconstruction, and is particularly suitable for the high reliability requirements of space-based on-orbit services.

[0024] Furthermore, by extracting multi-level features through a deep convolutional backbone network to construct an image pyramid, a comprehensive capture of information from low-level details to high-level semantic information is achieved, laying a feature foundation for accurate foreground segmentation. Through multi-scale Laplacian pyramid residual prediction and step-by-step reconstruction, high-precision saliency maps of the original resolution are generated while preserving target edge details, solving the problems of blurred edges and loss of small structures in traditional segmentation algorithms. By multiplying the binarized mask pixel by pixel with the original image, accurate removal of the space background is achieved while completely preserving all structural information of the space target.

[0025] Furthermore, cross-view feature matching establishes the correlation between multi-view images, providing basic data for camera pose determination; the combination of fundamental matrix / essential matrix / homography matrix and random sampling consensus algorithm filters out out-of-match points, solving the pose determination deviation problem caused by mismatch; incremental registration and triangulation enable the gradual expansion of the scene point set and accurate camera pose determination, adapting to multi-view observation modes such as space-based fly-around and escort flights; global nonlinear optimization is performed using bundle adjustment to minimize 3D point reprojection error and simultaneously optimize the coordinates of participating 3D points inside and outside the camera.

[0026] Furthermore, an end-to-end depth prediction pipeline consisting of "feature extraction - cost modeling - depth regression" was constructed, fully adaptable to multi-view systems. Figure Three The geometric principles of 3D reconstruction are established; through a specially designed multi-scale feature extraction structure, the feature representation capability of weak and non-textured regions of space-based targets is enhanced, solving the problem of insufficient feature extraction of weak texture regions by traditional algorithms; through homography transformation, the mapping of multi-view features in the depth dimension is realized, and the cost volume is constructed by combining view similarity calculation, establishing a deep fusion of multi-view geometric constraints and deep learning features; through 3D convolutional regularization and depth-weighted regression, a smooth and accurate dense depth map is output.

[0027] Furthermore, the iterative feature pyramid, through a bidirectional iterative fusion path, achieves full interaction between deep semantic information and shallow detail information, taking into account both the global structure and local detail features of the target; the dilated spatial convolutional pooling pyramid, through parallel convolution with multiple dilation rates and global pooling branches, captures multi-scale contextual information, solving the problem of poor feature adaptability caused by large changes in the size of space-based targets; the variable convolution dynamically adjusts the convolution sampling position by learning the offset, enabling the receptive field to adaptively focus on key areas such as target edges and corners, enhancing the feature capture capability for target geometric deformation and fine structures.

[0028] Furthermore, the core of multi-view stereo matching is to establish a cost distribution for the depth dimension through the feature similarity between views. This scheme uses a variance-based cost index, which measures the similarity between views by calculating the variance of multi-view features at the same spatial location. The smaller the variance, the higher the consistency of features between views, and the higher the confidence of the corresponding depth. This calculation method can adapt to any number of view inputs without requiring a fixed number of views, perfectly adapting to the flexible changes in the number of observation angles during space-based escort and fly-around operations. At the same time, the variance-based similarity calculation method has a strong anti-interference ability against feature fluctuations caused by changes in illumination and perspective, and can effectively filter feature noise caused by complex space-based illumination.

[0029] Furthermore, based on the confidence levels corresponding to different depths within the probability volume, depth value regression is achieved through the expectation calculation in the depth direction. Unlike the traditional argmax method that takes extreme values, the weighted average calculation effectively avoids discretization jumps in depth values, significantly improving the smoothness and continuity of the depth map. The formula clearly defines the upper and lower limits of the depth prediction range, allowing for dynamic adaptation to the observation distance range of space-based targets by combining camera parameters and sparse point clouds, avoiding computational interference from invalid depth intervals. Simultaneously, by using the probability value corresponding to each depth value as a weight, confidence-oriented depth regression is achieved, with high-confidence depth values ​​having a dominant weight, resulting in smoother depth results in low-confidence regions.

[0030] Furthermore, the loss function is designed for the network's multi-scale depth prediction structure. The L1 loss for each level's predicted depth and the actual depth is calculated separately, and then the total loss is obtained by weighting and summing these values ​​using preset weight coefficients, thus achieving multi-scale supervised training of the network. The L1 loss is less sensitive to outliers than the L2 loss, effectively reducing the interference of abnormal annotation values ​​caused by occlusion and sudden changes in illumination in space-based scenarios on network training, and improving the stability of the training process. The multi-scale weighted supervision method allows the network to learn both global structural features and local detail features simultaneously, solving the problem of global structural bias or loss of local details that easily occurs in single-scale supervision. The contribution of different scale losses can be flexibly adjusted through weight coefficients to adapt to the reconstruction needs of different structures of space-based targets.

[0031] Furthermore, by back-projecting pixel depth values ​​to the world coordinate system using camera geometry principles, a precise conversion from 2D depth maps to 3D point clouds was achieved, fully preserving the accuracy of depth prediction results. An innovative screening method combining photometric consistency and geometric consistency constraints was employed, performing multi-view cross-validation on candidate 3D points. Only 3D points that simultaneously satisfy both constraints across multiple views were retained, fundamentally filtering out noise points, mismatched points, and incorrectly reconstructed points. The photometric consistency constraint ensures the consistency of point appearance features across multiple views, while the geometric consistency constraint ensures the consistency of point spatial geometric position across multiple views. This dual constraint achieves a balance between accuracy and completeness.

[0032] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0033] In summary, this invention achieves high-precision dense reconstruction by removing background interference through saliency preprocessing, obtaining accurate pose using SfM, and combining multi-scale feature extraction with variance cost volume depth prediction. The dual consistency fusion strategy effectively reduces noise. This invention exhibits stronger robustness and higher accuracy under weak textures and complex lighting conditions, and is compatible with single-camera hardware, significantly reducing the hardware cost and computational complexity of on-orbit 3D reconstruction.

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0035] Figure One This is a flowchart of the three-dimensional reconstruction method of the present invention; Figure Two Flowchart of the incremental modeling method for motion recovery structures; Figure Three A schematic diagram of a sparse point cloud and camera pose of a space-based target to be reconstructed; Figure Four This is a schematic diagram of a deep learning-based multi-view depth prediction network structure. Figure Five This is a structural diagram of the space-based target model to be reconstructed. Figure Six This is a 3D point cloud diagram of the reconstructed space-based target model. Figure Seven A schematic diagram of a computer device provided in an embodiment of the present invention; Figure Eight This is a block diagram of a chip provided according to an embodiment of the present invention.

[0036] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0039] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0040] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0041] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0042] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0043] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0044] This invention provides a deep learning-based multi-view method for space-based targets. Figure Three The proposed reconstruction method employs background removal preprocessing based on saliency detection, which differs from existing preprocessing schemes for non-space-based scenarios. This addresses the inconsistency in geometric transformation between the target and background caused by complex space backgrounds, improving reconstruction accuracy from the source. It also utilizes a single-camera multi-view motion recovery structure, unlike existing monocular / binocular reconstruction schemes. This addresses the pain points of monocular systems failing to recover true scale, and binocular systems being cumbersome to calibrate and costly in terms of hardware, adapting to the lightweight deployment requirements of space-based on-orbit services. Furthermore, it designs an iterative feature pyramid cascaded dilated convolution and deformable convolution feature extraction structure, combined with variance cost volume depth prediction and dual-consistency point cloud fusion. This approach differs from traditional multi-view algorithms, addressing poor reconstruction of weakly textured regions and insufficient robustness in depth prediction, balancing accuracy and efficiency, and adapting to complex lighting and large-scale change scenarios in space.

[0045] Please see Figure One This invention provides a deep learning-based multi-view method for space-based targets. Figure Three The 3D reconstruction method includes the following steps: S1, Image Preprocessing Multi-view images of space-based targets are acquired using cameras, and background removal is performed using a salient target detection network.

[0046] The image to be processed is input into a pre-trained InSPyReNet salient object detection network model. The input image can be an RGB three-channel color image, and its size can be arbitrary; the model can adapt to this.

[0047] A deep convolutional neural network (such as Res2Net or Swing Transformer) is used to extract multi-level features from the input image. These features form an image pyramid that encompasses information from low-level details to high-level semantics.

[0048] The Laplacian pyramid is predicted at multiple scales. The Laplacian image stores the difference between the low-pass filtered signal and the original signal at each scale, i.e., high-frequency detail information. At each scale, the model predicts the Laplacian residual of the saliency map at that scale through a self-attention-based decoder.

[0049] Starting with the coarsest scale, the saliency map reconstructed from the previous level is combined with the Laplacian residual at the current scale to gradually reconstruct a saliency map at a finer scale. This process is repeated step by step until a high-precision saliency map at the original resolution is reconstructed.

[0050] The obtained high-precision saliency map is binarized to generate the final binary mask. In this mask, the foreground target area is white (pixel value 255), and the background area is black (pixel value 0). Multiplying this mask pixel-by-pixel with the original input image yields the foreground target image after removing the background.

[0051] Images with incorrect background removal or incomplete target acquisition should be manually removed.

[0052] Using a salient object detection network for background removal preprocessing of images can effectively eliminate interference from complex backgrounds in the space environment, avoid inconsistencies between the geometric transformation of space targets and backgrounds under dynamic conditions, and ensure the accuracy of reconstruction.

[0053] S2, Motion Recovery Structure Feature points are extracted and matched from the preprocessed image, and sparse point cloud and camera intrinsic and extrinsic parameters are obtained through incremental modeling.

[0054] Please see Figure Two and Figure Three Structure-of-motion reconstruction (SOR) is a method that uses a sufficient number of 2D images to obtain 3D information about the real scene relative to those images. The specific steps include: S201, Feature Extraction: This can be any highly specific feature, such as SIFT (COLMAP uses SIFT by default), mainly for subsequent feature matching. S202, Feature Matching: Matching each pair of input images (with very high time complexity) to obtain potential overlapping scenes; S203, Geometric Verification: There will inevitably be many outliers in the initial matching, so it is necessary to filter them out. Typically, the fundamental matrix (uncalibrated) / essential matrix (calibrated) and homography matrix (pure rotation / coplanar) are used, and mismatch points are removed using methods such as RANSAC to make the eigenvalue matrix more accurate.

[0055] For a calibrated camera, the essential matrix is ​​estimated, and its mathematical definition is: (1) in, and It represents the normalized coordinates of the matching points in the camera coordinate system. Essential matrix. It contains rotation and translation information.

[0056] For an uncalibrated camera, estimate the fundamental matrix: (2) in, It is the basic matrix.

[0057] For coplanar scenes or purely rotating cameras, the homography matrix can be estimated: (3) in, It is a homography matrix.

[0058] S204, Scene initialization: Select two non-panoramic views from all the multi-view perspectives as the starting positions; S205. Image registration: Estimate the pose of the image based on the existing model, obtain the 2D-3D match between the unknown image and the known image in the map based on the 2D-2D matching between the two, and then use RANSAC-PNP to solve the camera pose. S206, Triangulation: Expands the scene point set through triangulation, increasing scene coverage. This increases scene stability and enables registration of new images by providing additional 2D-3D correspondences. S207, BA Optimization: BA optimization is a large-scale nonlinear least squares optimization problem that aims to simultaneously optimize all camera parameters (pose, focal length, distortion coefficients, etc.) and all 3D point coordinates, minimizing the sum of squared errors of all 3D points reprojected onto their respective observed camera imaging planes. The objective function is: (4) in, Number of cameras; This represents the number of three-dimensional points. Let be the projection matrix of the i-th camera; Let J be the coordinates of the j-th 3D point; It is a binary variable, when the point by camera The value is 1 if observed, and 0 otherwise. The function represents the Euclidean distance between two-dimensional points x and y, i.e., the reprojection error.

[0059] By employing the structure-of-motion method, camera pose estimation is performed on images from the same camera at different viewpoints, reducing the increased costs associated with using multiple cameras and avoiding the cumbersome calibration process for binocular systems.

[0060] S3, Multi-view Depth Prediction The multi-view images and camera intrinsic and extrinsic parameters are used as inputs to the multi-view depth estimation network model, and the output is a dense depth map of multiple views.

[0061] S301, Multi-scale Feature Extraction of Images The input image is subjected to multi-scale deep feature extraction using iterative feature pyramids, dilated spatial convolutional pooling pyramids, and variable convolutional networks to obtain a series of feature maps.

[0062] The input image passes through an iterative feature pyramid structure, an ASPP module, and a deformable convolutional layer in sequence, and finally outputs a multi-scale feature map.

[0063] To capture information from images at different semantic levels, a multi-level feature pyramid was constructed. This pyramid does not simply process in parallel, but rather employs an iterative top-down and bottom-up fusion path to achieve full interaction between deep semantic information and shallow detail information.

[0064] The bottom-up path (encoder) uses a convolutional neural network as the backbone, downsampling the initial features multiple times to generate feature maps with decreasing scale but increasing channel number at three resolutions. The top-down path (decoder) starts with the deepest small-scale features and enlarges their size through upsampling operations (bilinear interpolation or transposed convolution). The upsampled features are then fused with corresponding features of the same spatial scale from the bottom-up path. Specifically, a 1×1 convolution is first performed to adjust the number of channels, and then element-wise added to the upsampled features.

[0065] A dilated spatial convolutional pooling pyramid (ASPP) module is introduced after a specific level of the feature pyramid. This ASPP module employs multiple dilated convolutional layers with different dilation rates in parallel, convolving the input features at different sampling densities: dilation rates of 1, 6, 12, and 18. Simultaneously, the ASPP module also includes a global average pooling branch. The output features of all parallel branches are concatenated along the channel dimension and then fused and dimensionality-reduced using a 1×1 convolution, outputting a feature map with strong multi-scale contextual representation capabilities. This process can be represented as: (5) in, This is the feature map after feature pyramid processing; Void ratio; This is a convolution operation; This is a global average pooling operation.

[0066] To adaptively adjust the spatial position of convolutional sampling, enabling the network to better adapt to the geometric transformations of different objects, this implementation integrates a deformable convolutional layer at the end of the feature extraction network. The deformable convolution dynamically adjusts the position of each sampling point by learning a two-dimensional offset associated with the input features. (6) in, It is a spatial location on the output feature map; and These represent the input and output features, respectively. Convolution weights; offsets Features are learned from the previous layer through an additional convolutional layer. This allows the network's receptive field to adaptively focus on more information-rich regions in the image (such as object edges and corners) rather than adhering to regular grid points, thus greatly enhancing the representational power of the features.

[0067] S302, Cost Body Construction and Regularization The depth prediction range is obtained from the camera parameters. The feature maps from different viewpoints are transformed to the depth direction of a certain viewpoint through homography on different depth planes. Similarity is calculated to obtain the cost volume, and the number of cost volume channels is reduced by 3D convolution.

[0068] After obtaining the image features, a 3D cost volume is constructed based on the extracted feature maps and camera parameters. A homography matrix is ​​established and a differentiable homography transformation is performed. The homography matrix can be represented as: (7) in, For depth value, Indicates the main axis direction of the reference camera. For camera internal parameters, , This refers to the camera's external parameters.

[0069] When aggregating multiple feature volumes into a single cost volume C, a variance-based cost metric is used to measure the similarity between views in order to accommodate any number of view inputs. The mapping relationship defined by the cost metric can be expressed as: (8) in, The volume dimensions of the feature body; The number of views participating in the aggregation.

[0070] The initial cost volume obtained at this point may contain noise, and smoothing is needed for further depth map prediction. Therefore, 3D CNN regularization is applied to the initial cost volume to obtain the probability volume. The probability volume can be used for pixel-by-pixel depth prediction and can also measure the confidence of the estimated value.

[0071] S303, Deep Regression The depth map for each viewpoint is obtained by weighted averaging based on the different confidence levels at different depths in the cost volume.

[0072] The depth map needs to be recovered from the probability volume. The expectation is calculated in the depth direction, and the weighted sum of all assumed depth values ​​is: (9) in, For depth value The probability value estimated for the location. The probability distribution along the depth direction can also reflect the quality of the depth estimation and acts as a threshold parameter in the subsequent outlier removal process.

[0073] During network training, the loss function is designed as an L1 loss between each predicted depth and the corresponding true depth value. Meanwhile, since the network has a three-layer depth prediction structure, the final loss value is a weighted sum of the three-layer depth losses.

[0074] (10) (11) in, This represents the actual depth value. To predict depth values; For a single pixel; These are the weighting coefficients for each layer; This represents the number of layers in the network structure; the maximum value is typically set to 3.

[0075] Please see Figure FourThe network takes multi-view preprocessed images as input and the overall process consists of three stages: feature extraction, depth prediction, and loss calculation. In the feature extraction stage, the input image first undergoes an iterative feature pyramid structure to achieve preliminary multi-scale feature fusion, and then sequentially passes through module A (Spatial Convolutional Pooling Module ASPP) and module D (Deformable Convolutional Module) to complete multi-scale deep feature extraction, corresponding to the implementation process of step S301. In the depth prediction stage, the extracted multi-view feature maps are mapped to different depth planes through module W (Differentiable Homography Transformation), and then the feature similarity between views and cost volume construction are completed through module M (Cost Volume Aggregation). After 3D convolution regularization, the dense depth map corresponding to the view is output through module R (Depth Regression), corresponding to the implementation processes of steps S302 and S303. In the network training stage, the predicted depth maps depth1, depth2, and depth3 output by the three levels are compared with the corresponding real depth maps gt1, gt2, and gt3 to calculate L1 losses Loss1, Loss2, and Loss3 respectively. Finally, the total loss is obtained by weighted summation with preset weights, realizing end-to-end supervised training of the network.

[0076] A deep learning-based multi-view depth prediction network is used to predict the depth of images from different perspectives, which improves the feature extraction capability for weak and textureless regions of spatial targets. By constructing cost volumes on different depth planes, the accuracy and smoothness of depth prediction are improved.

[0077] S4 and depth maps are merged into a 3D point cloud. By fusing camera parameters from different views with the corresponding depth maps, a 3D point cloud reconstruction result is obtained.

[0078] For each depth map, each pixel and its depth value are back-projected into the world coordinate system using camera geometry to generate candidate 3D points. By traversing all valid pixels in the depth map (i.e., points with depth values ​​within a reasonable range), an initial 3D point cloud corresponding to the view can be generated.

[0079] By fusing depth maps into 3D point clouds and constraining geometric and photometric consistency, the accuracy and completeness of the 3D point clouds are ensured with low computational cost.

[0080] In another embodiment of the present invention, a deep learning-based multi-view method for space-based targets is provided. Figure Three A 3D reconstruction system, which can be used to achieve the aforementioned deep learning-based multi-view reconstruction of space-based targets. Figure Seven Dimensional reconstruction method.

[0081] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. Please refer to [link to relevant documentation]. Figure Three The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, it implements the deep learning-based multi-view space-based target multi-view implementation in this embodiment. Figure Seven To avoid repetition, the 3D reconstruction methods will not be described in detail here. Alternatively, when the computer program 63 is executed by the processor 61, it implements the deep learning-based multi-view space-based target multi-view implementation of the embodiment. Figure Eight To avoid repetition, the functions of each model / unit in the dimensional reconstruction system will not be described in detail here.

[0082] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure One This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0083] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0084] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0085] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0086] Please see Figure Three The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0087] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure One The steps are shown in the figure.

[0088] Storage unit 620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include read-only memory (ROM) 6203.

[0089] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0090] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0091] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0092] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0093] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0094] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0095] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the deep learning-based multi-view of space-based targets in the above embodiments. Figure Five The corresponding steps of the dimensional reconstruction method. The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0096] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0097] This invention employs multi-view Figure Five Consistency checks, as the core filtering method, only retain three-dimensional points that are consistent across multiple perspectives.

[0098] This invention employs two consistency metrics for joint filtering: Photometric consistency check: For a candidate 3D point generated from a view, it is reprojected into another adjacent view, and its photometric error in that view is calculated. If the error is less than a preset threshold, the point is considered photometrically consistent between the two views. Geometric consistency check: After reprojecting the candidate 3D point into another view, a depth estimate is obtained. This estimate is compared with the actual value at that location in the depth map of that view. If the error is less than a preset threshold, the point is considered geometrically consistent between the two views. After these steps, a high-precision, highly consistent, and low-noise 3D point cloud model is finally output.

[0099] To quantitatively verify the technical advantages of the method of this invention, a control experiment was set up in this simulation experiment to compare its performance with existing mainstream 3D reconstruction methods. The specific experimental setup and results are as follows: Experimental Environment and Dataset Setup Hardware environment: CPU is Intel Xeon 8375C, GPU is NVIDIA RTX 6000 Ada, RAM is 64GB; Software environment: PyTorch 2.0 deep learning framework, CUDA 11.8 parallel computing architecture; Test dataset: based on Figure Six The on-orbit satellite model was used to generate three test datasets through Blender rendering, covering real-world space-based operating conditions: Condition 1 (Standard Condition): 49 evenly distributed observation angles around the flyby, uniform lighting, no strong backlighting or specular reflection; Case 2 (Weak Texture Challenge): With the same 49 observation views, the surface texture contrast of the model is reduced by 90% to simulate the weak texture characteristics of space-based targets; Condition 3 (Complex Illumination Condition): Similar to the 49 observation angles, but with the addition of strong backlight, specular reflection, and drastic changes in illumination, simulating the real illumination environment of space-based environments.

[0100] Comparison method: ① Traditional multi-view stereo matching algorithm COLMAP; ②The classic deep learning multi-view reconstruction algorithm MVSNet; ③The method of the present invention.

[0101] Evaluation metrics: Commonly used quantitative metrics in the field of 3D reconstruction are adopted, including Chamfer distance (CD, unit: mm, the smaller the value, the higher the reconstruction accuracy), F1 score (threshold 1mm, the higher the value, the better the overall performance), reconstruction integrity (%, the higher the value, the more complete the target structure is restored), and average time for single-frame depth prediction (ms, the smaller the value, the higher the inference efficiency).

[0102] In terms of reconstruction accuracy, the Chamfer distance of this invention is much lower than that of the two comparison algorithms under all working conditions. Under standard working conditions, it is only 0.42mm, which is 82.3% and 60.0% lower than COLMAP and MVSNet, respectively. Under complex lighting conditions, it is still only 0.91mm, showing a significant accuracy advantage. In terms of overall performance and reconstruction integrity, the F1 score and reconstruction integrity of this invention are the highest under all conditions, reaching 98.73% and 99.15% respectively under standard conditions. It still significantly outperforms the comparison algorithm under weak texture and complex lighting conditions, and has strong adaptability to extreme space-based scenarios. In terms of inference efficiency, the single-frame depth prediction time of this invention is stable at 72-74ms, which is more than 60% lower than COLMAP and about 24% lower than MVSNet, achieving better running efficiency while improving accuracy.

[0103] Experimental Results and Advantages First, the reconstruction accuracy and completeness are significantly improved. Under standard operating conditions, the Chamfer distance of the method of this invention is only 0.42mm, which is 82.3% lower than the traditional COLMAP algorithm and 60.0% lower than the MVSNet algorithm; the F1 score reaches 98.73%, and the reconstruction completeness reaches 99.15%, which can completely restore the fine structure and surface morphology of space-based targets, solving the core problems of insufficient reconstruction accuracy and structural incompleteness of existing methods.

[0104] Second, it exhibits exceptional robustness in weak texture and complex lighting conditions. Under weak texture conditions, the Chamfer distance of the method in this invention is only 0.68 mm, a reduction of 88.5% compared to COLMAP and 75.4% compared to MVSNet. Under complex lighting conditions, the F1 score of this invention remains at 93.58%, significantly higher than COLMAP's 41.52% and MVSNet's 68.79%, fully validating the feature extraction structure and depth prediction scheme of this invention, which can effectively solve the reconstruction failure problem caused by weak texture and drastic lighting changes in space-based targets.

[0105] Third, the inference efficiency is better suited to the needs of space-based on-orbit deployment. The average time for single-frame depth prediction in this invention is only about 72ms, which is 61.5% lower than the COLMAP algorithm and 24.0% lower than the MVSNet algorithm. While improving reconstruction accuracy, it further reduces computation time, which can adapt to the computing power limitations and real-time requirements of space-based on-orbit platforms.

[0106] Fourth, it has better engineering adaptability. This invention achieves real-scale reconstruction through a single-camera multi-view scheme, eliminating the need for multi-camera calibration and large baseline hardware deployment. By removing space background interference through saliency preprocessing, the entire process can be adapted to real engineering scenarios such as space-based escort and fly-around observations, and has stronger on-orbit application value than existing methods.

[0107] Please see Figure ThreeThis is a structural diagram of the space-based target model to be reconstructed used in this simulation experiment. The target is a typical on-orbit communication satellite model, and the model is a 1:1 reproduction of the actual structure, size, and surface physical characteristics of the space-based target. Specifically, the core of the model is a rectangular sealed payload bay with external dimensions of 1.2m × 0.8m × 0.6m. The outer surface of the payload bay is made of aerospace-grade smooth aluminum alloy with no obvious texture features, belonging to the typical weak texture and textureless reconstruction area in the space-based reconstruction scenario. Two sets of deployable solar panels are symmetrically arranged on the left and right sides of the model. The deployed size of a single solar panel is 2.4m × 0.9m. The solar panel substrate is a uniform matte triple-junction gallium arsenide coating surface with only a few reinforcing ribs at the edges, which is prone to depth estimation discontinuities. A 0.5m diameter parabolic communication antenna is installed on the top of the model. The curved structure is prone to local obstruction during multi-view observation. The bottom of the model is equipped with a φ300mm satellite-rocket docking ring and four sets of attitude control nozzles, which contain a large number of millimeter-level fine structural features, requiring extremely high reconstruction accuracy. This model fully covers the core technical challenges faced in the 3D reconstruction of space-based targets, such as weakly textured surfaces, complex curved surfaces, small features, and multi-component occlusion, and can comprehensively verify the engineering adaptability and reconstruction performance of the method of this invention.

[0108] Please see Figure One Figure 5 shows a 3D point cloud image obtained after 3D reconstruction of the space-based target model shown in Figure 5 using the method of this invention. From the visualization of the reconstruction results, the 3D point cloud output by the method of this invention completely restores the main payload bay structure of the satellite model. The solar panels on both sides have no structural defects or geometric deformations. The curved surface morphology of the parabolic antenna on top is accurately restored without surface collapse or insufficient smoothness. Millimeter-level fine structures such as the bottom docking ring and nozzles are all completely reconstructed. The point cloud is uniformly and densely distributed, with no obvious noise points or outliers. The weakly textured payload bay surface and solar panel substrate have no reconstruction holes or depth faults. The target edge contours are clear and sharp, without blurring or ghosting. The structure, size, and morphology are highly consistent with the original target model shown in Figure 5, thus achieving high-precision 3D reconstruction of the space-based target.

[0109] In summary, this invention provides a deep learning-based multi-view method for space-based targets. Figure OneThis paper presents a 3D reconstruction method and system that addresses the pain points of 3D reconstruction of space-based targets. It achieves high-precision, robust, and lightweight end-to-end 3D reconstruction. First, it effectively eliminates interference from complex space backgrounds through salient target detection preprocessing, avoiding inconsistencies in target and background geometric transformations and ensuring reconstruction accuracy from the source. Second, it employs a single-camera multi-view motion recovery structure scheme, eliminating the need for multi-camera systems and cumbersome calibration, thus restoring the true scale of the target and significantly reducing hardware costs and on-orbit deployment difficulty. Third, its innovative multi-scale feature extraction and depth prediction architecture significantly improves reconstruction performance in weak texture and complex lighting scenarios. Experiments show that its Chamfer distance is reduced by up to 88.5% compared to mainstream algorithms, with reconstruction integrity exceeding 99%. Fourth, its point cloud fusion scheme with dual consistency constraints balances point cloud accuracy and integrity with low computational cost, improving inference efficiency by more than 24% compared to mainstream solutions, demonstrating excellent value for space-based on-orbit engineering applications.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0113] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random-access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure One One or more processes and / or boxes Figure One A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure One One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0120] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A deep learning-based multi-view 3D reconstruction method for space-based targets, characterized in that, Includes the following steps: S1. Obtain the original images of the same spatial target from multiple observation perspectives, input the original images into a salient target detection network to obtain a foreground mask, and remove the background from the original images based on the foreground mask to obtain a preprocessed multi-view image. S2. Perform motion reconstruction processing on the preprocessed multi-view image. The motion reconstruction processing includes feature point extraction, cross-view feature matching, geometric consistency verification, incremental image registration, triangulation and bundling adjustment to obtain sparse 3D point cloud and camera intrinsic and extrinsic parameters corresponding to each view. S3. Input the preprocessed multi-view image and its corresponding camera intrinsic and extrinsic parameters into the multi-view depth estimation network to perform dense depth prediction for each view and output dense depth maps for each view. S4. Based on the camera parameters of each view and the corresponding dense depth map, back projection is performed to generate candidate 3D points. Based on the photometric consistency constraint and geometric consistency constraint, the candidate 3D points are jointly screened and multi-view fused to output the 3D point cloud reconstruction result of the spatial target.

2. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 1, characterized in that, In step S1, the salient object detection network is an InSPyReNet network, and the foreground mask generation and background removal processing specifically include: The original images from multiple observation perspectives are input into the pre-trained InSPyReNet network. The deep convolutional backbone network extracts multi-level features from the input images and constructs an image pyramid that covers low-level details to high-level semantic information. The Laplacian pyramid residuals of the corresponding saliency maps are predicted at multiple scales. Starting from the coarsest scale, the saliency maps reconstructed from the previous level are combined with the Laplacian residuals at the current scale to reconstruct a high-precision saliency map with the original resolution level by level. The high-precision saliency map is binarized to generate a binary foreground mask. The foreground mask is then multiplied pixel by pixel with the original input image to obtain a preprocessed multi-view image after removing the background.

3. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 1, characterized in that, The motion recovery structure processing in step S2 specifically includes: S201. Extract feature points from the preprocessed multi-view images and perform feature matching between pairs of multi-view images to obtain overlapping areas of scenes and matching point pairs. S202. Using the fundamental matrix, essential matrix, or homography matrix, combined with the random sampling consensus algorithm, filter out outliers in feature matching and retain matching point pairs that meet geometric consistency. S203. Select two non-panoramic views to complete scene initialization. Based on the existing matching point pairs and scene model, complete the image registration of the remaining views through the PNP algorithm, and expand the scene's 3D point set through triangulation. S204. Construct a nonlinear least squares optimization problem using the bundle adjustment method. Simultaneously optimize the camera intrinsic parameters, camera extrinsic parameters, and 3D point coordinates of all views. Minimize the sum of squared errors of all 3D points reprojected onto the corresponding view imaging plane. Finally, output the optimized sparse 3D point cloud, camera intrinsic parameters, and camera extrinsic parameters of each view.

4. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 1, characterized in that, The dense depth prediction in step S3 specifically includes the following steps: S301. Using iterative feature pyramid, dilated spatial convolutional pooling pyramid and variable convolutional network, multi-scale deep feature extraction is performed on the input preprocessed multi-view image to obtain multi-scale feature maps corresponding to each view. S302. Obtain the depth prediction range based on the camera intra-camera parameters and camera extrinsic parameters of each view. Map the feature maps of different viewpoints to the depth direction of the reference viewpoint through homography transformation on different depth planes. Construct a cost volume through view similarity calculation and perform regularization processing on the cost volume through three-dimensional convolution to obtain the depth probability volume. S303. Based on the confidence levels corresponding to different depths in the depth probability volume, perform a weighted average regression along the depth direction to obtain the dense depth map of the corresponding view.

5. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 4, characterized in that, In step S301, the iterative feature pyramid includes a bottom-up encoder path and a top-down decoder path. The encoder path downsamples the initial features multiple times using a convolutional neural network to generate feature maps with decreasing scale and increasing channel number at various resolutions. The decoder path starts from the deepest small-scale features, expands the feature size through upsampling, and then adds and fuses the upsampled features with the same spatial scale in the encoder path element by element after channel adjustment, thereby realizing bidirectional iterative interaction between deep semantic information and shallow detail information. The dilated spatial convolutional pooling pyramid module sets up multiple dilated convolutional layers with different dilation rates in parallel, as well as a global average pooling branch. After concatenating the output features of all parallel branches in the channel dimension, it is fused and dimensionality reduced by 1×1 convolution to output a feature map with multi-scale contextual representation capabilities. The variable convolutional network learns two-dimensional offsets from the features of the previous layer through additional convolutional layers, and dynamically adjusts the spatial position of the convolutional sampling according to the two-dimensional offsets, so that the receptive field of the network adaptively focuses on the target edges and corner regions of the image.

6. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 4, characterized in that, In step S302, the feature similarity between different views is calculated using a variance-based cost index to construct an initial cost body; the mapping relationship of the variance-based cost index is as follows: in, The volume dimensions of the feature body; The number of views participating in the aggregation. This represents the number of effective pixels.

7. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 4, characterized in that, In step S303, a weighted sum of all assumed depth values ​​is calculated along the depth direction to obtain the depth value of the corresponding pixel. The calculation formula is as follows: in, For depth value The estimated probability value of the location. This represents the maximum value of the depth prediction range. This is the minimum value of the depth prediction range.

8. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 4, characterized in that, During the training of the multi-view depth estimation network, the loss function used is a weighted sum of L1 losses from the multi-scale depth prediction results. The expression is: in, The number of views participating in the aggregation; The layer number of the network depth prediction structure; These are the weighting coefficients for each layer; This represents the number of layers in the network structure.

9. The deep learning-based multi-view 3D reconstruction method for space-based targets according to claim 1, characterized in that, Step S4 specifically includes: S401. For the dense depth map of each view, the effective pixels with depth values ​​within a reasonable range and their depth values ​​are back-projected to the world coordinate system using camera geometry principles to generate candidate 3D point clouds for the corresponding view. S402. Perform a multi-view consistency check on the candidate 3D point cloud, reproject the candidate 3D point to the adjacent view. If the point satisfies both the photometric consistency constraint and the geometric consistency constraint, it is retained; otherwise, it is discarded. The photometric consistency constraint is that the photometric error after reprojection of the candidate 3D point is less than a preset threshold. The geometric consistency constraint is that the error between the depth value after reprojection of the candidate 3D point and the depth value at that position in the dense depth map of the corresponding view is less than a preset threshold. S403. Output the filtered and fused 3D point cloud as the 3D reconstruction result of the space-based target.

10. A deep learning-based multi-view 3D reconstruction system for space-based targets, characterized in that, include: The preprocessing module is used to acquire the original images of the same spatial target from multiple viewing angles, input the original images into the salient target detection network to obtain the foreground mask, and remove the background of the original images based on the foreground mask to output the preprocessed multi-view images. The recovery module is used to perform motion recovery structure processing on the preprocessed multi-view image and output sparse 3D point cloud as well as the camera intra-camera parameters and camera extrinsic parameters corresponding to each view. The prediction module is used to input the preprocessed multi-view image and its corresponding camera intrinsic and extrinsic parameters into the multi-view depth estimation network, perform dense depth prediction for each view, and output dense depth maps for each view. The reconstruction module is used to back-project the camera parameters of each view with the corresponding dense depth map to generate candidate 3D points, and to perform joint screening and multi-view fusion of the candidate 3D points based on photometric consistency constraints and geometric consistency constraints, and output the 3D point cloud reconstruction result of the spatial target.

Images