A method for measuring a rock on Mars based on a semi-sphere projection model

Abstract

The application discloses a kind of Mars rock measurement methods based on hemispherical projection model, belong to the field of scene reconstruction;Specifically, first, the surface of Mars is photographed using binocular camera, modeling is carried out from camera principle and Mars car mechanical principle respectively, the corresponding relationship between the pixel points of each image and the real surface of Mars is established;And pose information of camera in Mars global coordinate system;Then, the intrinsic parameters of each image are analyzed, the distortion is corrected, and the left and right image pairs of binocular camera are corresponded, and the disparity map is calculated based on the matching feature points.Next, the rock outline in the existing Mars surface image segmentation dataset is used to determine the pixel position and distribution edge of the rock in the current disparity map, and the current disparity map is optimized using the disparity filtering algorithm;At the same time, combined with the pose of different cameras, the distance and position of the repeatedly appearing rock are calculated, and the distribution of the rock in the current disparity map is determined by verifying each other.The application improves the disparity estimation accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of scene reconstruction, and relates to technologies such as stereo image matching, three-dimensional scene modeling and Mars rock detection, and specifically relates to a Mars rock measurement method based on a hemispherical projection model. BACKGROUND

[0002] Mars is the most similar planet to the Earth, and is the focus of human deep space exploration. Many Mars exploration missions have collected Mars-related data sets. In the Mars scene, rocks are one of the main objects and are widely distributed on the surface of Mars. The detection and measurement of Mars rocks are important foundations of the Mars exploration program, and determining the size, distance and distribution of Mars rocks is one of the prerequisites for Mars ground exploration, which can provide support and support for tasks such as lander landing, Mars rover driving and path selection. On the other hand, the information of the rocks helps to judge the evolution process of the Mars address, and can provide data for the study of water resource distribution and geographical changes, and provide support for determining the evolution process of the Mars surface environment.

[0003] The existing measurement method mainly focuses on a single feature of the Mars scene, and lacks verification between data. The Mars rock detection task is usually based on remote sensing data or ground data. Due to the resolution limit, it is difficult to detect the detailed features of the Mars surface by remote sensing data, and the measurement of the Mars rock is mainly based on the ground data. However, the ground data is mainly collected by the load camera, and most Mars exploration missions are equipped with binocular cameras, and the Mars ranging method based on binocular stereo images is the main method to determine the unknown features of the Mars surface. However, the existing binocular disparity matching algorithms such as BM, SGB and the like are not sufficient to support the high-precision Mars rock measurement requirement.

[0004] In the Mars large scene, it is a challenge to determine the positional relationship between rocks and rocks, and the study of the size and distribution relationship is a blank in the related technology. The complexity of the Mars scene leads to an irregular sampling mode of the camera, and the collected data is discrete, so it is difficult to restore the size and position of the Mars rock in a unified scene. On the other hand, the accuracy of the existing technology in the Mars rock measurement is not high, and the performance needs to be improved in the face of complex Mars texture, repeated texture or non-ideal lighting and the like, and the corresponding measurement technology is not optimized for the rock, resulting in that the measurement accuracy of the rock area part does not meet the task requirement.

[0005] The prior art does not measure the Mars rock from the mode of a large scene, and does not optimize the rock. The method in document [1] studies the image matching in the navigation terrain reconstruction work of the survey probe, and improves the dynamic programming matching algorithm for the terrain reconstruction in the lunar environment, but it cannot meet the demand of rock measurement in the Mars scene. The method proposed in document [2] uses multi-source remote sensing image data of the Mars orbiter to construct a fine modeling and automatic classification of the Mars surface topography, combines photogrammetry and shading recovery shape method, studies and produces a high-resolution three-dimensional terrain of the landing area of Tianwen-1, but does not measure the rock, and the resolution is insufficient to meet the demand of the ground task.

[0006] In summary, the existing technology does not propose a targeted technical route or solution for "Mars rock measurement", resulting in the need to improve the measurement result of the Mars rock obtained by the general technology such as stereo matching.

[0007] [1] Li M L, Liu S C, Peng S. Improved dynamic programming in the lunar terrain reconstruction [J]. Opto-Electron Eng, 2013, 40: 6-11

[0008] [2] Liu S C, Tong X H, Liu S J, et al. Topography modeling, mapping and analysis of China's first Mars mission Tianwen-1 landing area from remote sensing images [J]. Journal of Deep Space Exploration, 2022, 9(3): 338-347. SUMMARY

[0009] The application proposes a Mars rock measurement method based on a hemispherical projection model, which focuses on the continuity and globality of Mars features, uses a projection model (PM) to establish a Mars global coordinate system to support high-precision, pixel-level Mars rock positioning, optimizes the parallax through a parallax filtering algorithm WLS to improve the accuracy of parallax estimation, extracts Mars features in a weak texture scene by combining a semantic segmentation algorithm, and thus restores the Mars features in a three-dimensional environment, having the advantage of better precision.

[0010] The Mars rock measurement method based on the hemispherical projection model comprises the following steps:

[0011] Step one, use binocular camera to take pictures of the surface of Mars, each object corresponds to two images at the same time;

[0012] Step two, analyze each image, take the camera optical axis as the shooting direction, calculate the corresponding position of each pixel in the three-dimensional scene of Mars within the field of view of the camera, and establish the corresponding relationship between the pixel point and the real surface of Mars;

[0013] For the target point P in the pixel coordinate system, the position is (u, v); Convert the projection of the image coordinate system as follows:

[0014]

[0015] (x i ,y i ) is the corresponding position of the target point P in the image coordinate system, where (u0, v0) is the image center point coordinate, (d x ,d y ) represents the size of the pixel on the camera photosensitive element.

[0016] Then, convert the image coordinate system to the camera coordinate system, which is represented as:

[0017]

[0018] (x c ,y c ,z c ) is the coordinate of the target point P in the camera coordinate system, and f is the focal length of the camera;

[0019] Finally, convert from the camera coordinate system to the world coordinate system, which is represented as:

[0020]

[0021] (x w ,y w ,z w ) is the coordinate of the target point P in the world coordinate system, where R is the rotation matrix, representing the rotation of the optical axis between the initial time and the world coordinate system, t is the translation vector from the origin of the world coordinate system to the camera optical center, 0 T is a three-dimensional column vector, R, t, which collectively describes the conversion relationship between the pixel point in the camera coordinate system and the world coordinate system.

[0022] Finally, combined with the camera parameters, the real position of the object in the scene of the surface of Mars in the current image is restored, and the pixel corresponding relationship is as follows:

[0023]

[0024] Step three, calculate the rotation matrix and translation vector of the camera, and obtain the pose information of the camera in the Mars global coordinate system;

[0025] The order of the Mars global coordinate system around the z-y-x axis is defined, and the rotation angles of the respective axes coincide with the body coordinate system of the Mars rover to obtain the rotation matrix R1, which is expressed as:

[0026]

[0027] Rotate around the z-axis by the yaw angle θ3, rotate around the y-axis by the pitch angle θ2, and rotate around the x-axis by the roll angle θ1;

[0028] Similarly, the rotation matrix R2 of the Mars rover body coordinate system rotating to the Mars rover mast coordinate system is obtained;

[0029] The rotation matrix R3 of the Mars rover mast coordinate system rotating to the Mars rover camera platform coordinate system is obtained;

[0030] The camera pose is obtained by combining the three rotation matrices R1, R2, and R3, and the odometer and size of the Mars rover to obtain the pose information of the camera in the Mars global coordinate system.

[0031] The translation vector t from the origin of the Mars coordinate system to the center of the camera is:

[0032] t = t1 + R1t2

[0033] t1 is the vector from the origin of the Mars coordinate system to the center of the rover, and t2 is the vector from the origin of the rover coordinate system to the center of the camera;

[0034] Step four, for each image that has established a corresponding relationship between the pixel points and the real Mars surface, correct the distortion of each image by analyzing the image intrinsic parameters, and select the left and right images of the same object to perform line correspondence to make them located on the same plane.

[0035] Specifically:

[0036] First, re-project the pixels according to the camera focal length, imaging origin, and distortion coefficient intrinsic parameters, etc. to solve the distortion error caused by the camera lens, and then correct it;

[0037] After correcting the left and right images of the same object, perform image alignment according to the rotation matrix and translation vector between the two image pairs of external parameters, so that the epipolar lines of the two images are exactly on the same horizontal line.

[0038] Step five, for the left and right images located on the same plane, use stereo vision to match the matching feature points in the overlapping area of the two images, and calculate the disparity map based on the matching feature points.

[0039] The stereovision matching detection uses scale invariant feature transform (SIFT), speeded up robust features (SURF) and fast rotation orientation (ORB) and other feature descriptors to respectively extract feature points in the image, and performs feature point matching.

[0040] The corrected left and right images are taken as inputs of the SGBM algorithm, and a disparity map is determined according to the binocular camera data, and mapping points P on the left and right image planes are respectively represented by P left and P right . The disparity d is the difference between the column coordinates of P left and P right , and is solved iteratively.

[0041] Step six, in the existing Mars surface image segmentation data set, the rock contour is detected by using the semantic segmentation method, and the pixel position and distribution edge of the rock in the current disparity map are determined;

[0042] First, the rock is recognized and segmented, and according to the characteristics of different rocks, the semantic segmentation model based on deep learning, the shadow method or the manual marking method is used for rock segmentation, so as to obtain the range of rock pixel distribution and generate a mask. The selected way of the mask is to select the rock area pixels in the image, and to obtain the target area image by shielding.

[0043] According to the rock mask obtained by semantic segmentation, the pixel distribution range of the rock area in the image is determined, and the disparity of the rock area in the disparity map is obtained.

[0044] Step seven, according to the position of the rock pixel, based on the imaging principle of the binocular camera and the triangulation principle, the position and size of the pixel are calculated; the current disparity map is optimized by using the disparity filtering algorithm;

[0045] According to the position of the rock pixel, the distribution direction and the distribution boundary of the corresponding pixel in the disparity map are found, and the weighted least square filtering algorithm is used to filter the disparity map.

[0046] Given a disparity map g with a size of n*m, and a filtered disparity map u, then the loss function is:

[0047]

[0048] Where A x , A y are diagonal matrices with a x , a y as diagonal elements, d' x , d' y are forward difference matrices, and are backward difference operators. λ is a scale factor, and the larger λ is, the stronger the smoothing effect is.

[0049] Step eight, taking the pose information of the camera in the global coordinate system of Mars as the reference, combining the positions of different cameras, the distance and position of the repeatedly appearing rocks are calculated, and the distribution of the rocks in the current parallax map is determined by verification.

[0050] The depth Z of the rock to the camera is calculated according to the principle of similar triangles, and the formula is as follows:

[0051]

[0052] b is the baseline of the binocular camera, that is, the distance between the optical centers of the left and right cameras; f is the focal length of the camera.

[0053] According to the edge information of the semantic segmentation mask, the size of the rock is calculated.

[0054] The advantages of the present application are:

[0055] 1) A Mars rock measurement method based on a hemispherical projection model, which is used for the camera pose description of Tianwen No. 1 terrain navigation camera of the Mars rover. The correspondence between the data and the scene is restored by modeling the pose of the navigation camera, the position of the rock in the three-dimensional scene is determined, and the demand for feature positioning and measurement under large scale is met.

[0056] 2) A Mars rock measurement method based on a hemispherical projection model, which proposes an external parameter acquisition method based on a hemispherical projection model. The image can be preprocessed in a large amount of data, the position relationship between the images is obtained, the image is preliminarily corrected and the feature points are matched, and the binocular camera data with good matching degree is obtained.

[0057] 3) A Mars rock measurement method based on a hemispherical projection model, which designs a parallax matching and filtering module suitable for Mars rocks, which can accurately measure the distance and size of the Mars rocks under large scale scene, improves the accuracy of related measurement, and restores the distribution of the Mars rocks in multiple scenes, providing support for Mars surface exploration. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of a Mars rock measurement method based on a hemispherical projection model of the present application;

[0059] Figure 2 A schematic diagram of modeling from the mechanical principle of the Mars rover of the present application;

[0060] Figure 3 A conversion schematic diagram of pixels from the image coordinate system, the camera coordinate system and the world coordinate system of the present application;

[0061] Figure 4 A schematic diagram of a hemispherical model established by modeling from the mechanical principle of the Mars rover of the present application;

[0062] Figure 5 Fig. 1 is a schematic diagram of view difference based on binocular camera matching data according to the present application. DETAILED DESCRIPTION

[0063] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0064] In order to obtain the distribution information of rocks in the Mars scene, the existing technology does not consider information extraction from the global perspective, and the present application provides a Mars rock measurement method based on a hemispherical projection model. The hemispherical projection model can obtain global information, and the filtering algorithm can improve the matching accuracy. Specifically, by focusing on the continuity and globality of the Mars features, a projection model (PM) is used to establish a Mars global coordinate system, and according to the position and attitude of the camera, the rotation angle of the mast, the position and attitude of the Mars rover and other information, the Mars rocks in multiple scenes can be described in the Mars global coordinate system to support high-precision, pixel-level Mars rock positioning. The SGBM algorithm is used for disparity matching, and the disparity is optimized by the disparity filtering algorithm WLS to improve the accuracy of disparity estimation. Combined with the semantic segmentation algorithm, the Mars features in the weak texture scene are extracted, so that the Mars features are restored in the three-dimensional environment, which has scientific research significance for supporting Mars surface exploration.

[0065] The Mars rock measurement method based on the hemispherical projection model, as shown in Figure 1 The specific steps are as follows:

[0066] Step one, use a binocular camera to take pictures of the Mars surface, and each object corresponds to two images of left and right at each time;

[0067] Step two, analyze each image taken respectively, and use the hemispherical projection model to determine the corresponding relationship between the pixel points and the real Mars surface;

[0068] This step models from the camera principle and mechanical principle, and the specific process is as follows:

[0069] Step 101, based on the parameters of the navigation terrain camera, the spatial position and attitude angle information of the camera during image analysis are obtained. An analysis method is proposed for the mechanical structure of the Zhurong rover and the camera mounting method;

[0070] As shown in Figure 2As shown, when collecting the image of Mars, the camera, the camera platform, the mast and the rover body have certain rotation and displacement. When the Zhurong rover works, first, the mast mechanical arm is unfolded to the vertical position, then the gimbal is tilted up and down, and finally the mechanical arm is yawed left and right to complete the sequence imaging of the scientific exploration point. In order to describe the imaging process of the current image, the imaging pose of the current camera needs to be determined according to the current rover coordinates, the mast angle, the camera height position and the camera shooting direction.

[0071] In step 102, the image is projected by using the extracted camera information, the projection source is determined according to the navigation terrain camera position, and the shooting range of the camera at the position is calculated;

[0072] As Figure 3 shown, (u, v) is the corresponding position of the target point P in the pixel coordinate system, (x i ,y i ) is the corresponding position of the target point P in the image coordinate system, wherein (u0, v0) is the principal point of the image center, (d x ,d y ) represents the size of the pixel on the photosensitive element of the navigation terrain camera. The projection from the pixel coordinate system to the image coordinate system can be represented as:

[0073]

[0074] (x c , y c , z c ) is the coordinate of point P in the camera coordinate system, and f is the focal length of the navigation terrain camera. The conversion from the image coordinate system to the camera coordinate system can be represented as:

[0075]

[0076] (x w , y w , z w ) is the coordinate of point P in the world coordinate system, and the transformation from the camera coordinate system to the world coordinate system is represented as:

[0077]

[0078] wherein R is a rotation matrix representing the rotation of the optical axis in the world coordinate system between the initial time, t is the translation vector of the world coordinate system origin to the camera optical center, 0 T is a three-dimensional column vector, and R and t collectively describe the conversion relationship between the pixel point in the camera coordinate system and the world coordinate system.

[0079] The imaging range of the camera in the field of view of the navigation terrain camera is calculated in the direction of the optical axis of the terrain navigation camera, the corresponding position of the pixel in the three-dimensional scene of Mars is calculated, and the corresponding relationship between the pixel and the real surface of Mars is established.

[0080] In step 103, the pixel is converted from the pixel coordinate system to the image coordinate system, the camera coordinate system and the world coordinate system in sequence combined with the camera parameters, the real position of the object in the scene on the surface of Mars in the image is restored, and the corresponding relationship of the pixel is as follows:

[0081]

[0082] In step 104, each pixel in the image is projected into the world coordinate system, and the approximate range of the image distribution and whether there is overlap between different images are determined by the projection.

[0083] In step three, the rotation matrix and the translation vector of the camera are calculated, and the pose information of the camera in the global coordinate system of Mars is obtained, which provides the initial position for stereo vision matching;

[0084] This step models the mechanical principle of the Mars rover; calculates the rotation matrix R, translation vector t and other external parameters of the camera according to the semi-spherical projection model, and is used in stereo vision matching, based on the mechanical structure of Zhuxing and the camera installation method, a semi-spherical model is established, as shown in Figure 4 Specifically:

[0085] The order of the global coordinate system of Mars is defined around the z-y-x axis, and the rotation angle coincides with the body coordinate system of the Mars rover, the yaw angle is θ3 around the z axis, the pitch angle is θ2 around the y axis, and the roll angle is θ1 around the x axis, and the rotation matrix R1 is obtained, which is expressed as:

[0086]

[0087] Similarly, the rotation matrix R2 of the body coordinate system of the Mars rover rotating to the mast coordinate system of the Mars rover is obtained;

[0088] The rotation matrix R3 of the mast coordinate system of the Mars rover rotating to the camera platform coordinate system of the Mars rover is obtained;

[0089] The navigation terrain camera pose is obtained by combining the three rotation matrices R1, R2 and R3, and the pose information of the camera in the global coordinate system of Mars is obtained by combining the odometer and size of the Mars rover.

[0090] The translation vector t from the origin of the Mars coordinate system to the center of the camera is:

[0091] t=t1+R1t2

[0092] t1 is the vector from the origin of the Mars coordinate system to the center of the rover, and t2 is the vector from the origin of the rover coordinate system to the center of the camera; the rotation matrix of the rover coordinate system relative to the Mars coordinate system is R1.

[0093] Step 4: Using stereo vision matching, calculate the rotation matrix and translation vector between the binocular cameras based on the hemispherical projection model to calculate the extrinsic parameters of the image, analyze the intrinsic parameters of the image, correct image distortion, and make the left and right images lie on the same plane; use stereo matching to detect the relevant matching points of the left and right images, and calculate the disparity map based on the matching feature points.

[0094] For each image that establishes the correspondence between pixels and the real Martian surface, the distortion of each image is corrected by analyzing the image intrinsic parameters, and the left and right images of the same part are selected to be on the same plane.

[0095] After correction, the two images in a left-right image pair are on the same plane, but the relationship between different image pairs is not on the same straight line. This step first corrects each image, and then selects the binocular images for row correspondence; that is, the object of correction is each image, and the object of row correspondence is the selected binocular image pair.

[0096] Specifically:

[0097] First, the left and right images are corrected separately based on intrinsic parameters such as camera focal length, imaging origin, and distortion coefficients to eliminate distortion in the left and right views. Second, the binocular images are aligned using extrinsic parameters such as rotation matrix and translation vector between the left and right image pairs.

[0098] Based on the camera's intrinsic parameter matrix, pixels are reprojected to resolve distortion errors caused by the camera lens. Distortion is divided into radial distortion and tangential distortion. Radial distortion is caused by the lens shape and is the effect of the lens's own shape on light propagation. Tangential distortion is caused by the non-parallel position of the lens and the imaging plane, which will cause the position of the line when it passes through the lens and is projected onto the imaging plane to change.

[0099] By strictly aligning the two distortion-corrected images row-wise, ensuring that the epipolar lines of both images are on the same horizontal line, any point in one image will necessarily have the same row number as its corresponding point in the other image. A one-dimensional search within that row is sufficient to find the corresponding point. These steps ensure that the imaging origin coordinates of the left and right views of the camera are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned, reducing the computational process for parallax.

[0100] Step 5: For the left and right images located on the same plane, use stereo vision matching to detect feature points in the overlapping area of ​​the two images and perform matching. Calculate the disparity map based on the matched feature points.

[0101] The distortion of the image is restored by stereo rectification, so that the same points are basically on the same line; the feature points of the overlapping area of the two photos are detected and matched, and the feature descriptors such as scale-invariant feature transform (SIFT), speeded up robust features (SURF) and orientationally robust binary (ORB) are used to extract the feature points in the image respectively, and the feature point matching is performed.

[0102] Disparity map estimation based on binocular camera matching data, as shown in Figure 5

[0103] The distance between the optical centers of the binocular camera is the baseline b, and the left and right images are located in the O w -X w Y w Z w coordinate system. The optical axes Z left and Z right are parallel to each other, and the mapping points P on the left and right image planes are represented by P left and P right respectively. The disparity d is the difference between the column coordinates of P left and P right , and is solved iteratively.

[0104] The left and right images corrected by the semi-sphere projection model are used as the input of the SGBM algorithm, and the disparity map is determined according to the binocular camera data.

[0105] Step six, in the existing Mars surface image segmentation data set, the rock contour is detected by using the semantic segmentation method, and the pixel position and distribution edge of the rock in the current disparity map are determined;

[0106] First, the rock is identified and segmented, and according to the characteristics of different rocks, the semantic segmentation model based on deep learning, the shadow method or the manual marking method is used for rock segmentation, and the range of rock pixel distribution is obtained and the mask is generated. The selected mask is selected in the image, and the target area image is obtained by shielding the rock area pixels.

[0107] The method uses the Mars surface image segmentation data set TWMARS, which is a rock segmentation data set based on the "Tianwen-1" data set made by the team, which can provide basis for rock edge extraction and positioning in the disparity map.

[0108] According to the rock mask obtained by semantic segmentation, the pixel distribution range of the rock area in the image is determined, and the disparity of the rock area in the disparity map is obtained.

[0109] Step seven, according to the position of the rock pixel, based on the imaging principle of binocular camera and the principle of triangulation, the position and size of the pixel are calculated; the current disparity map is optimized by using the disparity filtering algorithm; ​

[0110] According to the position of the rock pixel, the distribution direction of the corresponding pixel in the disparity map and its distribution boundary are found, and a weighted least square filtering algorithm is used to filter the disparity map. Given a disparity map g, the size of which is n*m, the filtered disparity map is u, and the loss function is:

[0111]

[0112] where A x , A y is a diagonal matrix with a x , a y as diagonal elements, d′ x , d′ y is a forward difference matrix, and is a backward difference operator; λ is a scale factor, and the larger λ is, the stronger the smoothing effect is.

[0113] The filtering makes the disparity of the rock part in the disparity map change more smoothly, and can better reflect the outline of the mask.

[0114] Step eight, taking the pose information of the camera in the global coordinate system of Mars as the reference, the distance and position of the repeatedly appearing rock are calculated in combination with the positions of different cameras, and are verified with each other to determine the distribution of the rock in the current disparity map.

[0115] Because the measurement result of the rock is relative to the current camera position, the relative positions of the rock in different images need to be calculated through the positions of the cameras.

[0116] Based on the imaging principle and the triangulation principle of stereo vision, the distance and size of the rock are calculated according to the baseline, focal length and other information of the binocular camera. The disparity d between the left and right photos is inversely proportional to the depth Z, and according to the similarity principle of triangle, the following relationship can be established:

[0117]

[0118] Wherein, according to the disparity d, the baseline b and the focal length f, the depth Z from the rock to the camera can be calculated. Z is the distance from the pixel to the center of the navigation terrain. According to the edge information of the semantic segmentation mask, the size of the rock is calculated.

[0119] The present application can effectively obtain the rock distribution information in a large scene, efficiently and quickly correct the binocular data and other preprocessing work, and optimize the matching accuracy of the rock part to obtain more accurate rock size and distance information.

[0120] Firstly, the camera pose is modeled by using the semi-sphere projection model, and the corresponding relationship between the pixel point and the real Mars surface is determined to determine the approximate range of the image distribution and whether there is overlap between different images.

[0121] Specifically, the image of Zhurong navigation terrain camera is analyzed to obtain the spatial position, attitude angle and other information of the camera when the image is taken; the extracted information is used to project the image to establish the corresponding relationship between the pixel and the real Mars surface. Combined with the camera parameters, the pixel coordinate system, image coordinate system, camera coordinate system and world coordinate system are converted in turn to restore the real position of the object in the Mars surface scene in the recovered image; each pixel in the image is projected into the world coordinate system to determine the position and coverage range of the Mars image through projection.

[0122] Then, the rotation matrix and translation vector between the binocular cameras are calculated based on the hemispherical projection model to calculate the extrinsic parameters of the image, the image intrinsic parameters are analyzed, the image distortion is corrected to make the left and right images located on the same plane; the related matching points of the left and right images are detected by stereo matching, and the disparity map is calculated based on the matching feature points.

[0123] Specifically, the distortion of the recovered image is corrected by stereo correction to make the same points basically on the same line; the feature points in the overlapping area of the two photos are detected and matched, the feature descriptors such as scale invariant feature transform (SIFT), speeded up robust features (SURF) and orientationally fast rotation (ORB) are used to extract the feature points in the image respectively, and the feature point matching is performed. Disparity map estimation is performed based on binocular camera matching data, in this stage, the image corrected by the hemispherical projection model is taken as the input data of SGBM algorithm, and the disparity map is determined according to the binocular camera data. The innovation is to use the proposed hemispherical projection model to obtain the external parameters of the camera, to pre-process the image, and to obtain the binocular camera data with better matching degree.

[0124] Finally, the rock outline detected by the method of semantic segmentation is combined to determine the pixel position and distribution edge of the rock in the disparity map; according to the position of the rock pixel, the imaging principle of the binocular camera and the principle of triangulation, the position and size of the pixel are calculated; the disparity map is optimized by using disparity filtering algorithm; combined with the positions of different cameras, the distance and position of the repeatedly appearing rocks are calculated, and they are verified with each other to determine the distribution of the rocks.

[0125] Specifically: first, the rock is identified and segmented, and according to the characteristics of different rocks, a semantic segmentation model based on deep learning, a shadow method, or a manual labeling method is used for rock segmentation to obtain the range of rock pixel distribution and generate a mask. According to the position of the rock pixels, the distribution direction and its distribution boundary of the corresponding pixels in the disparity map are found, and a weighted least squares filtering algorithm is used to filter the disparity map, so that the disparity of the rock part in the disparity map is relatively smooth, and the outline of the mask is better. Based on the imaging principle and triangulation principle of stereo vision, the distance and size of the rock are calculated according to the baseline, focal length and other information of the binocular camera. Combined with the positions of different cameras, the distance and position of the repeatedly appearing rock are calculated, and verified with each other to determine the distribution of the rock. The innovation is that a disparity filtering module is designed according to the characteristics of the Mars rock data, which improves the measurement accuracy of the rock.

[0126] Embodiment:

[0127] Step 1: Based on C++ language and OpenCV library, the original Mars data is parsed on a computer device to extract the navigation terrain camera frame data parameters, including shooting time, shooting position, shooting attitude angle (yaw, pitch, roll), camera parameters (focal length, pixel size, photosensitive element size, optical axis direction), rotation angle, etc.

[0128] Step 2: The parameters extracted by the camera are projected, and a hemispherical projection model is established according to the shooting position of the Mars rover, the rotation direction of the Mars rover, the attitude angle of the mast, the pitch angle of the camera platform, the size of the Mars rover, the installation structure and other information. Calculate the image coverage range and project it onto the Mars surface model.

[0129] Step 3: According to the hemispherical projection model, the rotation matrix, translation vector and other external parameter information between the binocular camera data are calculated, and the image is corrected so that the same feature points are in the same horizontal position on the corresponding image.

[0130] Step 4: According to the principle of stereo disparity matching, the disparity map of the binocular data is calculated, the SGBM algorithm is used to match the feature points, the parallax distance of the feature points is obtained, and the image disparity is calculated.

[0131] Step 5: According to the rock identification and segmentation results, the rock part in the disparity map is optimized, the least weighted least squares filtering algorithm is used to optimize the disparity map, the image edge information is saved, and the rock area pixels are smoothed.

[0132] Step 6: Based on the principle of triangulation, the position and size of the rock are calculated, and the position between different cameras is verified.

[0133] As shown in Table 1, the distance of rock c~f measured by the method is 9.877 m, 6.377 m, 5.208 m and 10.820 m respectively, which is consistent with the mainstream measurement method and measurement result. It can be seen from the table that the measurement accuracy of the application is millimeter, which is better than the current measurement method.

[0134] Table 1

[0135]

[0136] As shown in Table 2, the size of rock c measured by the method is (3.470 m, 1.130 m), the size of rock d is (0.110 m, 0.144 m), and the size of rock e is (0.178 m, 0.109 m), which is consistent with the mainstream measurement method and measurement result. It can be seen from the table that the measurement accuracy of the application is millimeter, which is better than the current measurement method.

[0137] Table 2

[0138]

Claims

1. A method of measuring a rock on Mars based on a semi-sphere projection model, characterized by, The method comprises the following steps: Step one, use binocular camera to take pictures of the surface of Mars, each object corresponds to two images at the same time; Step two, analyze each image, calculate the corresponding position of each pixel in the three-dimensional scene of Mars in the field of view of the camera, and establish the corresponding relationship between the pixel and the real surface of Mars; Specifically: For the target point P in the pixel coordinate system, the position is (u, v); convert it to the projection of the image coordinate system as follows: (x i ,y i ) is the corresponding position of the target point P in the image coordinate system, where (u0,v0) is the image center point coordinate, (d x ,d y ) represents the size of the pixel on the camera photosensitive element; Then, convert the image coordinate system to the camera coordinate system, represented as: (x c , y c , z c ) are coordinates of the target point P in the camera coordinate system, and f is the focal length of the camera. Finally, convert from the camera coordinate system to the world coordinate system, represented as: (x w , y w , z w ) are coordinates of the target point P in the world coordinate system, where R is a rotation matrix of the optical axis between the world coordinate system and the camera coordinate, t is a translation vector of the world coordinate system origin to the camera optical center, 0 T is a three-dimensional column vector, R, t collectively describe the conversion relationship between the pixel point in the camera coordinate system and the world coordinate system; Finally, combine the camera parameters to restore the real position of the object in the scene of the surface of Mars in the current image, and the pixel correspondence is as follows: Step three, calculate the rotation matrix and translation vector of the camera to obtain the pose information of the camera in the global coordinate system of Mars; Define the order of the global coordinate system of Mars around the z-y-x axis, and the rotation angles of the three axes are the same as the body coordinate system of the Mars rover, to obtain the rotation matrix R1 represented as: Rotate around the z axis by the yaw angle θ3, rotate around the y axis by the pitch angle θ2, and rotate around the x axis by the roll angle θ1; Similarly, obtain the rotation matrix R2 of the body coordinate system of the Mars rover rotating to the mast coordinate system of the Mars rover; The rotation matrix R3 of the mast coordinate system of the Mars rover rotating to the camera platform coordinate system of the Mars rover; Combine the three rotation matrices R1, R2, and R3 to obtain the camera pose, and combine the odometer and size of the Mars rover to obtain the pose information of the camera in the global coordinate system of Mars; The translation vector t from the origin of the global coordinate system of Mars to the center of the camera is: t=t1+R1t2 t1 is the vector from the origin of the global coordinate system of Mars to the center of the body coordinate system of the Mars rover, and t2 is the vector from the origin of the body coordinate system of the Mars rover to the center of the camera coordinate system; Step four, for each image in which the corresponding relationship between the pixel and the real surface of Mars is established, correct the distortion of each image by analyzing the image intrinsic parameters, and select the left and right images of the same object to perform line correspondence to make them located on the same plane; Step five, for the left and right images located on the same plane after correction, use stereo vision to match the matching feature points in the overlapping area of the two images, and calculate the disparity map based on the matching feature points; The corrected left and right images are taken as inputs of the SGBM algorithm, and a disparity map is determined according to the binocular camera data, mapping points P on the left and right image planes are respectively represented by P left and P right , and a disparity d is a difference between column coordinates of P left and P right , and is solved iteratively; Step six, in the existing Mars surface image segmentation data set, use the semantic segmentation method to detect the rock outline, and determine the pixel position and distribution edge of the rock in the current disparity map; First, identify and segment the rock, and use the semantic segmentation model based on deep learning, the shadow method, or the manual marking method according to the characteristics of different rocks to segment the rock, obtain the range of rock pixel distribution, and generate a mask; According to the rock mask obtained by semantic segmentation, determine the pixel distribution range of the rock region in the image, and obtain the disparity of the rock region in the disparity map; Step seven, according to the rock pixel position determined in the view difference, based on the imaging principle of the binocular camera and the triangulation principle, calculate the size of the rock in the view difference; use the disparity filtering algorithm to optimize the current disparity map; Step eight, taking the pose information of the camera in the global coordinate system of Mars as the reference, combining the positions of different cameras, the distance and three-dimensional position of the repeatedly appearing rocks are calculated, and the real rock distribution in the current parallax map is determined by verifying each other; The depth Z of the rock to the camera is calculated according to the principle of similar triangles, and the formula is as follows: b is the baseline of the binocular camera, that is, the distance between the optical centers of the left and right cameras; f is the focal length of the camera.

2. The method of claim 1, wherein the method is based on a semi-spherical projection model. The step four is specifically: first, the pixel is re-projected according to the camera focal length, imaging origin and distortion coefficient internal parameter data, the distortion error caused by the camera lens is solved, and then the correction is carried out. After the left and right images of the same object are corrected, the image is aligned according to the rotation matrix and translation vector external parameter data between the two image pairs, so that the epipolar lines of the two images are exactly on the same horizontal line.

3. The method of claim 1, wherein the method is based on a semi-spherical projection model. In step five, the scale invariant feature transform, accelerated robust feature and fast rotation directional feature descriptor are used for respectively extracting the feature points in the image, and the feature point matching is carried out.

4. The method of claim 1, wherein the method is based on a semi-spherical projection model. In step seven, according to the position of the rock pixel, the distribution direction and the distribution boundary of the corresponding pixel in the parallax map are found, and the weighted least square filtering algorithm is used for filtering the parallax map. Given the parallax map g, the size of which is n*m, and the filtered parallax map u, the loss function is: where A x , A y is a diagonal matrix with a x , a y as diagonal elements, d′ x , d′ y is a forward difference matrix, and is a backward difference operator; λ is a scale factor, the larger λ is, the stronger the smoothing effect is.

Images