A method and system for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching

Abstract

The present invention relates to a method and system for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching, and the method comprises the following steps: S1, acquiring binocular images of spatial non-cooperative targets and performing correction; S2, using binocular images of spatial non-cooperative targets to calculate a disparity map of spatial non-cooperative targets based on unsupervised deep learning of a binocular stereo matching deep neural network model; S3, calculating point cloud data of spatial non-cooperative targets; S4, calculating a point cloud to be registered for spatial non-cooperative targets, and initializing a reference point cloud of spatial non-cooperative targets; S5, performing registration using an ICP algorithm, and calculating rotation angles and translation amounts; S6, adding new point cloud data in the current binocular image to the reference point cloud as a new reference point cloud for subsequent registration; S7, adding new point cloud data while calculating the pose to complete three-dimensional reconstruction. The beneficial effect is that the pose estimation and three-dimensional reconstruction of spatial non-cooperative targets are performed accurately.

Description

[Technical field]

[0001] The present invention relates to the technical field of spatial image processing, and in particular to a method and system for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching. [Background technology]

[0002] The iterative closest point method ICP (Iterative Closest Point) is based on the point set to point set (PSTPS) registration method, that is, the point set to point set registration method based on quaternions. After determining the corresponding nearest point set from the measured point set, the method proposed by Faugera and Hebert is used to calculate the new nearest point set; iterative calculation is performed until the objective function value composed of the residual sum of squares remains unchanged, and the iteration process ends. The ICP algorithm is mainly used to solve the registration problem based on free-form surfaces. Binocular stereo matching is a hot topic in binocular vision research. The binocular camera takes left and right viewpoint images of the same scene, and uses the stereo matching algorithm to obtain the disparity map, and then obtains the depth map. Binocular stereo matching can generally be divided into four steps: matching cost calculation, cost aggregation, disparity calculation and disparity optimization.

[0003] Deep learning (DL) is a research direction in the field of machine learning (ML). The introduction of machine learning makes it closer to its original goal - artificial intelligence (AI). Deep neural networks (DNN) are the basis of deep learning. DNN is sometimes also called multi-layer perceptron (MLP). According to the position of different layers of DNN, the neural network layers inside DNN can be divided into three categories: input layer, hidden layer and output layer. Supervised learning is a machine learning task that infers functions from labeled training data sets. Unsupervised learning solves various problems in pattern recognition based on training samples of unknown categories (not labeled).

[0004] In recent years, autonomous satellite on-orbit servicing has attracted increasing attention from all countries and has gradually become a research hotspot. At present, the on-orbit servicing technology for cooperative spacecraft is relatively mature. However, a considerable number of on-orbit servicing objects are non-cooperative targets, such as space debris, faulty satellites, rocket upper stages and non-cooperative spacecraft. Due to the non-cooperative nature and complexity of the targets and the limitations of existing technologies, there is still no precedent for fully autonomous on-orbit servicing of non-cooperative spacecraft. Research on autonomous on-orbit servicing technology for non-cooperative targets in space is of great significance and application value.

[0005] After search and analysis, the relevant patents of the prior art and their defects are as follows: Invention publication CN105976353A, invention name "Spatial non-cooperative target pose estimation method based on global matching of model and point cloud", uses the ICP algorithm for pose estimation, but the defect is that it requires the use of a depth camera and a known target model point cloud, which is often impossible in the case of non-cooperative targets, and does not include the function of three-dimensional reconstruction of non-cooperative targets; Invention publication CN106679634A, invention name "A spatial non-cooperative target pose measurement method based on stereo vision", uses stereo vision for pose estimation, but the defect is that it needs to extract edge information, the process is cumbersome, there are few points available for three-dimensional reconstruction, and the reconstructed model is inaccurate; Invention publication CN113313821A, invention name "A fast three-dimensional reconstruction system and method", reconstructs the target in three dimensions, but the defect is that it requires the use of a depth camera and multiple cameras.

[0006] In view of the above-mentioned defects of the prior art, the present invention makes technical improvements to the spatial non-cooperative target pose estimation and three-dimensional reconstruction. [Summary of the invention]

[0007] The purpose of the present invention is to provide a method for accurately performing spatial non-cooperative target pose estimation and three-dimensional reconstruction.

[0008] To achieve the above object, the technical solution adopted by the present invention is a spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching, comprising the following steps:

[0009] S1. In the process of the binocular camera flying around the non-cooperative target, the binocular images of the spatial non-cooperative target, i.e., the left image and the right image of the spatial non-cooperative target, are acquired by using the binocular camera, i.e., the left image and the right image of the spatial non-cooperative target, and the binocular images of the spatial non-cooperative target are corrected using the calibration data of the binocular camera;

[0010] S2, based on unsupervised deep learning of the binocular stereo matching deep neural network model, the spatial non-cooperative target binocular image obtained in step S1 is used to calculate the spatial non-cooperative target disparity map;

[0011] S3, calculating the spatial non-cooperative target point cloud data using the calibration parameters of the binocular camera and the spatial non-cooperative target disparity map obtained in step S2;

[0012] S4, based on the spatial non-cooperative target point cloud data in step S3, calculating the spatial non-cooperative target to-be-registered point cloud, and initializing the spatial non-cooperative target reference point cloud;

[0013] S5, using the ICP algorithm to register the spatial non-cooperative target to-be-registered point cloud and the spatial non-cooperative target reference point cloud obtained in step S4, obtain the current spatial non-cooperative target binocular image pose, and solve the spatial non-cooperative target rotation angle and translation from the current spatial non-cooperative target binocular image pose transformation matrix;

[0014] S6, according to the rotation angle and translation of the spatial non-cooperative target calculated in step S5, the new point cloud data in the current spatial non-cooperative target binocular image in step S4 is added to the spatial non-cooperative target reference point cloud as the new spatial non-cooperative target reference point cloud in the subsequent step S5 registration;

[0015] S7, repeating S4 to S6, adding new point cloud data to the spatial non-cooperative target reference point cloud while calculating the spatial non-cooperative target posture, to complete the three-dimensional reconstruction of the spatial non-cooperative target.

[0016] Preferably, the above-mentioned method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching further includes a binocular stereo matching deep neural network model pre-training step:

[0017] S01, creating a binocular stereo matching deep neural network model with binocular images as input and disparity maps as output;

[0018] S02. Use the spatial non-cooperative target public dataset to perform supervised deep learning to pre-train the binocular stereo matching deep neural network model.

[0019] Preferably, the above-mentioned method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching further includes a binocular stereo matching deep neural network model pre-training step:

[0020] S03. During use, the binocular stereo matching deep neural network model is trained so that the binocular stereo matching deep neural network model can adapt to the spatial environment and output a disparity map that is more accurate.

[0021] Preferably, in the above-mentioned method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching, if there is no spatial non-cooperative target reference point cloud: step S4 initializes the calculated spatial non-cooperative target point cloud to be registered as the spatial non-cooperative target reference point cloud.

[0022] Preferably, in the above-mentioned method for estimating the pose of a spatial non-cooperative target and reconstructing three-dimensional objects based on binocular stereo matching, the specific method for calculating the point cloud data of the spatial non-cooperative target in step S3 is as follows: Assume that the calibration parameter of the binocular camera is the pixel focal length (simple explanation of the meaning) f x 、f y and offset (simple explanation of meaning) c x 、cy , the baseline of the binocular camera (simple explanation) is b, and the disparity (simple explanation) corresponding to the point (u, v) in the left image and / or the right image in the binocular image is dis u，v , then the calculation formulas for the coordinates (x, y, z) of the point (u, v) in space are:

[0023]

[0024]

[0025]

[0026] Another object of the present invention is to provide a system for accurately performing spatial non-cooperative target pose estimation and three-dimensional reconstruction.

[0027] To achieve the above-mentioned further purpose, the technical solution adopted by the present invention is a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching, including a binocular camera, computing resources and a spatial non-cooperative target, the binocular camera including a left camera and a right camera; the binocular camera and the computing resources are carried by a spacecraft and operate in a space orbit; and are used to execute the above-mentioned spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching.

[0028] Preferably, the left camera and the right camera have completely the same parameters, are connected to the computing resource, and are located on the left and right sides of the computing resource.

[0029] Preferably, the left camera and the right camera adopt Super 16mm lens with a focal length of 50mm, and the captured image resolution is set to 1366*768; the left camera and the right camera are 0.5m apart, facing in exactly the same direction and perpendicular to the line connecting the left camera and the right camera.

[0030] Preferably, the binocular camera is 10-100m away from the non-cooperative target in space.

[0031] The present invention provides a method and system for spatial non-cooperative target pose estimation and 3D reconstruction based on binocular stereo matching, which has the following beneficial effects: it has the following advantages: 1. It only needs a pair of binocular cameras, and does not need a depth camera to obtain dense depth information of the non-cooperative target; 2. The system workflow is simple and clear; 3. It does not need a known non-cooperative target model point cloud, but uses a method of gradually fusing reference point clouds to perform 3D reconstruction of the non-cooperative target; it has the following innovations: 1. It uses unsupervised deep learning to obtain a disparity map in space, so that the neural network adapts to the spatial environment and obtains an accurate disparity map; 2. It does not need a known non-cooperative target model point cloud; 3. While performing non-cooperative target pose estimation, it performs 3D reconstruction of the non-cooperative target by gradually fusing reference point clouds.

Brief Description of the Drawings

[0032] Figure 1 It is a schematic diagram of the structure of a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching.

[0033] Figure 2 The present invention is a flow chart of a method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching. [Specific implementation method]

[0034] The present invention will be further described below in conjunction with embodiments and with reference to the accompanying drawings.

[0035] Example 1

[0036] This embodiment implements a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching.

[0037] Figure 1 This is a schematic diagram of the structure of a spatial non-cooperative target pose estimation and 3D reconstruction system based on binocular stereo matching. Figure 1 As shown, in this embodiment, a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching is composed of binocular cameras (left camera and right camera) and computing resources.

[0038] The left and right cameras have the same parameters, are connected to the computing resources, and are located on the left and right sides respectively. In the simulation environment, the binocular camera uses a Super 16mm lens with a focal length of 50mm, and the image resolution is set to 1366*768. The left and right cameras are 0.5m apart, facing the same direction, and perpendicular to the line connecting the left and right cameras.

[0039] During use, the spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching in this embodiment is 10-100m away from the non-cooperative target; the camera is kept shooting the non-cooperative target, and the non-cooperative target is flew around, and the pose solution and three-dimensional reconstruction are completed during the flight around the non-cooperative target.

[0040] In this embodiment, a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching is usually carried by a spacecraft and operates in orbit. The binocular camera takes images of non-cooperative targets such as faulty satellites in the camera's field of view to obtain left and right images, respectively, and uses computing resources to perform binocular stereo matching to complete non-cooperative target pose solution and three-dimensional reconstruction.

[0041] Example 2

[0042] This embodiment implements a spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching, based on Example 1, a spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching.

[0043] Figure 2 This is a flow chart of a method for spatial non-cooperative target pose estimation and 3D reconstruction based on binocular stereo matching. Figure 2 As shown, the present embodiment provides a method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching, comprising the following steps:

[0044] Step 1: using a binocular camera to obtain a binocular image of a non-cooperative target in space, and using the calibration data of the binocular camera to calibrate the binocular image;

[0045] Step 2: Use the unsupervised binocular stereo matching algorithm based on deep learning to calculate the disparity map using the binocular image obtained in step 1. The specific process is as follows:

[0046] Step 2-1, create a deep neural network model, the input is a binocular image, and the output is a disparity map;

[0047] Step 2-2, design a supervised loss function and use a public dataset to pre-train the deep neural network created in step 2-1;

[0048] Step 2-3, design an unsupervised loss function, use the deep neural network pre-trained in step 2-2 in the spatial environment, adapt it to the spatial environment during use, and make the output disparity map more accurate;

[0049] Step 3: Use the calibration parameters of the binocular camera and the disparity map obtained in step 2 to calculate the point cloud data of the spatial non-cooperative target as the reference point cloud in subsequent steps; the specific method for calculating the point cloud data of the spatial non-cooperative target is as follows:

[0050] Assume that the camera calibration parameter is pixel focal length f x 、f y and offset c x 、c y , the baseline of the binocular camera is b, and the disparity corresponding to the point (u, v) in the left image (or right image) of the binocular image is dis u，v , then the calculation formulas for the coordinates (x, y, z) of the point (u, v) in space are:

[0051]

[0052]

[0053]

[0054] Step 4, using the method from step 1 to step 3, while the binocular camera flies around the non-cooperative target with the satellite, continue to collect binocular images and calculate the point cloud to be registered;

[0055] Step 5, using the ICP algorithm to align the point cloud to be registered in step 4 and the reference point cloud in step 3, obtain the pose of the current frame, and solve the rotation angle and translation from the pose transformation matrix;

[0056] Step 6: According to the rotation angle and translation calculated in step 5, the point cloud of the current frame in step 4 is added to the reference point cloud as the reference point cloud for subsequent point cloud matching;

[0057] Step 7, repeat steps 4 to 6, add new points to the reference point cloud while calculating the pose of the spatial non-cooperative target, and complete the three-dimensional reconstruction.

[0058] The advantages of the spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching in this embodiment are: only a pair of binocular cameras is needed, and no depth camera is needed to obtain dense depth information of the non-cooperative target; the system workflow is simple and clear; there is no need to know the non-cooperative target model point cloud, but the non-cooperative target is reconstructed in three dimensions by gradually fusing the reference point cloud.

[0059] The innovative points of the spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching in this embodiment are as follows: unsupervised deep learning is used to obtain the disparity map in space, so that the neural network adapts to the spatial environment and obtains an accurate disparity map; no known non-cooperative target model point cloud is required; while estimating the non-cooperative target pose, the non-cooperative target is three-dimensionally reconstructed by gradually fusing the reference point cloud.

[0060] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching, characterized in that The following steps are involved: S1. In the process of the binocular camera flying around the non-cooperative target, the binocular images of the spatial non-cooperative target, i.e., the left image and the right image of the spatial non-cooperative target, are acquired by using the binocular camera, i.e., the left image and the right image of the spatial non-cooperative target, and the binocular images of the spatial non-cooperative target are corrected using the calibration data of the binocular camera; S2, based on unsupervised deep learning of the binocular stereo matching deep neural network model, the spatial non-cooperative target binocular image obtained in step S1 is used to calculate the spatial non-cooperative target disparity map; S3, calculating the spatial non-cooperative target point cloud data using the calibration parameters of the binocular camera and the spatial non-cooperative target disparity map obtained in step S2; S4, based on the spatial non-cooperative target point cloud data in step S3, calculating the spatial non-cooperative target to-be-registered point cloud, and initializing the spatial non-cooperative target reference point cloud; S5, using the ICP algorithm to register the spatial non-cooperative target to-be-registered point cloud and the spatial non-cooperative target reference point cloud obtained in step S4, obtain the current spatial non-cooperative target binocular image pose, and solve the spatial non-cooperative target rotation angle and translation from the current spatial non-cooperative target binocular image pose transformation matrix; S6, according to the rotation angle and translation of the spatial non-cooperative target calculated in step S5, the new point cloud data in the current spatial non-cooperative target binocular image in step S4 is added to the spatial non-cooperative target reference point cloud as the new spatial non-cooperative target reference point cloud in the subsequent step S5 registration; S7, repeating S4 to S6, adding new point cloud data to the spatial non-cooperative target reference point cloud while calculating the spatial non-cooperative target posture, to complete the three-dimensional reconstruction of the spatial non-cooperative target.

2. The method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching according to claim 1, characterized in that It also includes the pre-training steps of the binocular stereo matching deep neural network model: S01, creating a binocular stereo matching deep neural network model with binocular images as input and disparity maps as output; S02. Use the spatial non-cooperative target public dataset to perform supervised deep learning to pre-train the binocular stereo matching deep neural network model.

3. The method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching according to claim 2 is characterized in that It also includes the pre-training steps of the binocular stereo matching deep neural network model: S03. During use, the binocular stereo matching deep neural network model is trained so that the binocular stereo matching deep neural network model can adapt to the spatial environment and output a disparity map that is more accurate.

4. The method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching according to claim 1, characterized in that If there is no spatial non-cooperative target reference point cloud: step S4 initializes the calculated spatial non-cooperative target to-be-registered point cloud as the spatial non-cooperative target reference point cloud.

5. The method for spatial non-cooperative target pose estimation and three-dimensional reconstruction based on binocular stereo matching according to claim 1, characterized in that Step S3 calculates the point cloud data of the non-cooperative target in space. The specific method is as follows: Assume that the calibration parameter of the binocular camera is the pixel focal length f x 、f y and offset c x 、c y , the baseline of the stereo camera is b, and the disparity corresponding to the point (u, v) in the left image and / or the right image in the stereo image is dis u，v , then the calculation formulas for the coordinates (x, y, z) of the point (u, v) in space are:

6. A spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching, characterized by: It includes a binocular camera, computing resources and a space non-cooperative target, wherein the binocular camera includes a left camera and a right camera; the binocular camera and the computing resources are carried by a spacecraft and run in a space orbit; Used to execute a spatial non-cooperative target pose estimation and three-dimensional reconstruction method based on binocular stereo matching as described in any one of claims 1 to 5.

7. The spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching according to claim 6, characterized in that: The left camera and the right camera have completely the same parameters, are connected to the computing resource, and are located on the left and right sides of the computing resource.

8. The spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching according to claim 7, characterized in that: The left camera and the right camera use Super 16mm lenses with a focal length of 50mm, and the captured image resolution is set to 1366*768; the left camera and the right camera are 0.5m apart, facing in exactly the same direction and perpendicular to the line connecting the left camera and the right camera.

9. The spatial non-cooperative target pose estimation and three-dimensional reconstruction system based on binocular stereo matching according to claim 8, characterized in that: The binocular camera is 10-100m away from the non-cooperative target in space.

Images