A biomimetic eye-based binocular active vision semi-dense depth estimation method

Abstract

A kind of binocular active vision semi-dense depth estimation method based on bionic eye, characterized in that, comprising the following steps: S100, image is collected using binocular bionic eye camera;S200, the image collected in S100 is input into SLAM system, and camera pose and road marker point coordinates are estimated using SLAM system to track key frame;S300, using the key frame pose and its corresponding key frame image data obtained in S200, through polar search, block matching is combined with triangulation, then semi-dense depth estimation is realized using depth filter.The application filters some image block gradient that does not meet the requirements of block matching NCC calculation in the process of block matching, so compared with dense direct method, the application can better achieve the requirement of real-time.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to problems such as bionic eye motion control, camera pose and landmark coordinate estimation optimization, and semi-dense depth estimation. Specifically, it relates to a binocular active vision semi-dense depth estimation method based on a bionic eye. Background Technology

[0002] In the field of robotics, the autonomous navigation capability of mobile robots in unknown environments has always been a highly anticipated and crucial research direction. SLAM technology provides a key solution for mobile robots to locate themselves and build maps in unknown environments. Binocular passive vision SLAM and monocular passive vision SLAM have been extensively studied in the current SLAM field. However, in outdoor and large-scale scenes with dynamic objects and uneven texture distribution, traditional passive vision SLAM still has many limitations, affecting the accuracy and robustness of the algorithm and even causing it to fail. This is because the camera cannot actively select the scene to capture, thus making it impossible to avoid the negative impact of areas with low texture distribution and dynamic objects on the SLAM algorithm.

[0003] Currently, mainstream feature-point visual SLAM only estimates the 3D positions of sparse feature points, resulting in a map containing only these sparse landmarks. While such sparse maps are sufficient for robot relocalization, their performance is limited for more demanding tasks such as navigation, obstacle avoidance, and scene 3D reconstruction. Therefore, the ability to perform semi-dense depth estimation is crucial for improving map usability and promoting the practical application of SLAM technology. Currently, visual SLAM primarily uses camera characteristics, deep learning, or direct methods for semi-dense or dense depth estimation. However, the most common application in related robots is still using RGB-D cameras to directly acquire the depth values ​​of all pixels in an image via TOF (Time of Flight) or structured light methods. This is because RGB-D cameras are relatively easy to use and can directly measure depth through hardware in the sensor, without consuming significant computational resources for estimation. However, dense mapping with RGB-D cameras is only suitable for specific scenarios. In outdoor or large-scale environments, RGB-D cameras cannot be well applied, and their power consumption and cost are high. Summary of the Invention

[0004] The purpose of this invention is to provide a method for estimating semi-dense depth using binocular active vision based on a bionic eye.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A binocular active vision semi-dense depth estimation method based on a bionic eye includes the following steps:

[0007] S100 uses a binocular bionic eye camera to acquire images;

[0008] S200 inputs the images acquired by S100 into the SLAM system, and uses the SLAM system to track key frames and estimate camera pose and landmark coordinates;

[0009] S300 utilizes the keyframe poses and corresponding keyframe image data acquired by S200, and achieves semi-dense depth estimation through epipolar search, block matching combined with triangulation, and then using a depth filter. Specifically:

[0010] S301, Select a frame that enters step S300 as a reference keyframe, and assume that the inverse depth of all pixels in the image data corresponding to the reference keyframe satisfies an initial Gaussian distribution to provide an initial value for subsequent depth information fusion.

[0011] S302, insert a new keyframe and determine the projection point position through epipolar search and block matching;

[0012] S303, calculate the depth of the triangulated 3D points and its uncertainty based on the geometric relationship between the reference keyframe and the current keyframe;

[0013] S304, integrate the Gaussian distribution of the inverse depth corresponding to the current observation depth into the previous estimate to establish the distribution of the fused inverse depth; if the uncertainty is less than a certain threshold, the depth data is considered to have converged and the 3D point depth is determined; otherwise, return to step S302 to continue information fusion.

[0014] Specifically, step S302 is as follows:

[0015] Let u1 be a pixel in the image corresponding to the reference keyframe, and let l2 be the epipolar line projecting the depth range corresponding to this pixel into the image corresponding to the current keyframe; let I be the image corresponding to the reference keyframe. k The image corresponding to the current keyframe is denoted as I. t The rotation matrices of the world coordinate system relative to the reference keyframe and the current keyframe are R and R, respectively. kW and R tW The translation vectors of the world coordinate system relative to the reference keyframe and the current keyframe are t and t, respectively. kW and t tW The intrinsic parameter K of the left camera was obtained through calibration. L The relationship between the reference keyframe and the current keyframe in pixel coordinates is as follows:

[0016]

[0017] Among them, u kTo reference a pixel in a keyframe, d k The depth of the spatial point corresponding to this pixel; when d is known k u k When u is available, it can be calculated. t Position; when given u k When two components are further increased by an increment du and dv, u can be calculated. t Incremental du t dv t This yields an affine transformation consisting of a linear relationship between the coordinate transformations of the keyframe and the current frame within a local area:

[0018]

[0019] Next, based on the affine transformation matrix, the camera rotation is taken into account in the block matching:

[0020] Take the pixel block surrounding u1 as At the same time, the pixel block on l2 is recorded as The correlation between pixel blocks surrounding u1 and pixel blocks on the epipolar line is compared using a mean-free normalized cross-correlation method. When the angle between the pixel block gradient and the epipolar line is large, the reliability of the block matching is low. Therefore, only pixel blocks whose pixel gradient is in the epipolar direction or within a certain threshold in the epipolar direction are matched. According to the definition of NCC, we have:

[0021]

[0022] After determining the step size, by traversing the NCC values ​​of the specified pixel blocks and the source pixel blocks on the corresponding epipolar lines, a non-convex NCC distribution function can be obtained. The point with the highest NCC is then used as the matching pixel block for the source pixel block.

[0023] In step S302, if the highest NCC value matched by the block is still lower than the threshold, it is considered that the epipolar search and block matching for the source pixel block in the image corresponding to the current key frame has failed. After entering a new key frame and the corresponding camera pose optimized by the SLAM system in S200, matching and information fusion convergence are performed again.

[0024] In step S303, the depth of the three-dimensional point is calculated using triangulation, and the calculation method is as follows:

[0025] Suppose there are two frames, I1 and I2, with I1 as the reference and the transformation matrix of I2 as T; the camera optical centers are O1 and O2; there is a feature point p1 in I1, and a corresponding feature point p2 in I2; connecting the optical center O1 with the feature point p1, and the optical center O2 with the feature point p2, we obtain rays O1p1 and O2p2; when the camera does not perform pure translational motion, the lines O1p1 and O2p2 intersect at a point P in the scene, which is the position of the map points corresponding to the two feature points p1 and p2 in the 3D scene;

[0026] Let x1 and x2 be the normalized coordinates corresponding to feature points p1 and p2, then their relationship satisfies:

[0027] s2x2=s1Rx1+t, where s1 and s2 are the depths of two feature points p1 and p2, respectively, and the rotation matrix R and translation vector t are known;

[0028] From a geometric perspective, we can find a three-dimensional point on ray O1p1 such that its projection position is close to p2, and similarly we can find it on ray O2p2.

[0029] Taking the calculation of s1 as an example, transforming the above equation yields an equation that only concerns s1: Solve the equations to directly obtain s1; similarly, obtain s2; at this point, the depths of the two feature points are preliminarily determined.

[0030] When O1p1 and O2p2 cannot intersect due to noise, the least squares method is used to solve the problem.

[0031] In step S303, as new current keyframes continuously appear, the corresponding pixel blocks are found using epipolar search and block matching, and the depth is estimated using triangulation. The observed depth in the reference keyframe is calculated and converted into inverse depth. This inverse depth also follows a Gaussian distribution.

[0032] in, The depth deviation is obtained by combining the pixel error after epipolar search with the depth deviation calculated from the geometric relationship of triangulation.

[0033] In step S100, a panoramic camera is used to guide the gaze of the binocular bionic eye camera. The steps include:

[0034] S101, acquire panoramic images, and then detect key scene information in the panoramic images that affect the execution of the SLAM task, including feature points and dynamic objects;

[0035] S102, assign values ​​to pixels in the panoramic image based on key scene information to construct a panoramic value image;

[0036] S103, Project the field of view areas of the left and right bionic eye cameras onto the panoramic value image respectively to obtain the left field of view projection area and the right field of view projection area; Simultaneously control the overlap of the left field of view projection area and the right field of view projection area; Merge the left field of view projection area and the right field of view projection area to obtain the current binocular field of view projection area;

[0037] S104: Obtain the average value of the current binocular field of view projection area in the panoramic value image; determine the value of the current binocular field of view projection area compared with the value threshold: if the average value of the current binocular field of view projection area is greater than the value threshold, do not move the current binocular field of view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system; if the average value of the current binocular field of view projection area is less than or equal to the value threshold, search for a target binocular field of view projection area with a higher value than the value threshold in the panoramic value image, and then proceed to step S105;

[0038] S105, calculate the required displacement of the left and right bionic eye cameras according to the current binocular field of view projection area and the target binocular field of view projection area; move the left and right bionic eye cameras according to the displacement amount; and finally use the high-value images collected by the left and right bionic eye cameras in the target binocular field of view projection area as the input of the SLAM system.

[0039] In step S104

[0040] By using a rectangle of the same size as the projected gaze area of ​​the current bionic eye camera as a sliding window within the gazeable region of the panoramic image, a search is performed from near to far, starting from the current gaze area with a certain step size s. This search is then combined with the panoramic image value to obtain the average value V of each window. m ;

[0041] Define the value threshold V th If the mean value of the currently viewed area is higher than the value threshold, the camera remains stationary; conversely, if the mean value of the currently viewed area is less than or equal to the value threshold, the gaze is shifted to an area where the mean value is higher than the value threshold. th And the area closest to the current gaze area.

[0042] In step S105, the rotational motion of the left bionic eye camera is represented by a rotation matrix R. LCLT The rotational motion of the right camera is represented by the rotation matrix R. RCRT R is solved using the following method. LCLT :

[0043] The center pixel coordinates u of the target gaze region in the panoramic value map PT According to the projection model The image is back-projected onto the projection sphere of the panoramic camera, and the projection point P on the sphere is also back-projected. PT Transform to the initial pose coordinate system O of the left camera LO Obtain the target optical axis direction P of the left camera LOT : The center point of the pixel plane of the area currently being gazed at by the left bionic eye camera. Projecting the model through the camera Back-projected onto the normalized plane, and then transformed into the initial pose coordinate system O of the left bionic eye camera. LO Obtain the left camera in the initial pose coordinate system O LO The current pose P in LOC : Therefore, the left bionic eye camera needs to be adjusted from the current pose P LOC Move to target pose P LOT ;

[0044] Calculate the current attitude P LOC With target attitude P LOT Rotation axis between:

[0045] According to the definition of dot product, for rotation angle θ LCLT :

[0046] The rotation matrix P can be obtained using the Rodriguez formula based on the rotation angle and rotation axis. LCLT :

[0047]

[0048] Similarly, solve R RCRT ;

[0049] By solving the rotation matrices of the left and right cameras, the rotation commands that the bionic eye control module needs to issue to the control motors of the left and right bionic eyes are obtained, so that the bionic eye cameras can shift their field of view to high-value areas in the scene and achieve gaze guidance.

[0050] The SLAM system adopts an improved SLAM algorithm based on the ORB-SLAM2 framework, and includes four modules: a tracking thread, a local mapping thread, a loop closure thread, and a global basis analysis (BA) thread.

[0051] (1) By matching feature points of the current frame with the local map, the tracking of each frame is realized. The camera pose is initialized and the local map is tracked using motion mode, PnP relocalization mode or reference key frame mode. The camera pose is estimated by minimizing the reprojection error using PnP. At the same time, key frames are judged and run as a tracking thread.

[0052] (2) Manage local maps and perform local BA optimization through local mapping threads;

[0053] (3) Use the DBoW2 library to detect large loops and correct the camera trajectory accumulation error by performing pose graph optimization, thereby forming a loop closure thread;

[0054] (4) After completing the pose graph optimization, start the global BA thread to obtain globally consistent camera pose and map structure.

[0055] In the SLAM system, if the current frame meets any of the following conditions, the current frame is set as a keyframe:

[0056] (1) The current frame is the first frame after the bionic eye gaze control reaches the target gaze area;

[0057] (2) More than 13 frames have passed since the last global relocation;

[0058] (3) The local mapping thread is in an idle state;

[0059] (4) More than 15 frames have passed since the last keyframe set;

[0060] (5) The translation distance between the current frame and the previously set keyframe exceeds the threshold t. th ;

[0061] (6) The number of successfully tracked feature points in the current frame reaches 70 or more;

[0062] (7) The number of successfully tracked feature points in the current frame is less than 85% of that in the reference keyframe.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) This invention is based on the visual mechanism of the human eye. When the feature points in a scene are unevenly distributed, it can construct a panoramic value map to assign high value to areas with relatively dense feature points, thereby guiding the bionic eye to move towards these areas. Dense feature points are beneficial for the pose estimation optimization of the bionic eye camera and the construction of a global map, improving the stability of the SLAM system. At the same time, it also provides a texture basis for the subsequent use of block matching for semi-dense depth estimation by the bionic eye camera.

[0065] (2) In constructing the panoramic value map, this invention eliminates the interference of dynamic objects on camera state estimation optimization and semi-dense depth estimation. If dynamic objects appear during the semi-dense depth estimation process, they will affect the block matching process. After the dynamic object moves, the pixels that should have been matched on the corresponding epipolar line disappear, thus matching incorrect pixel blocks and causing matching failure or depth estimation error. Therefore, the step of eliminating dynamic objects provides better quality image data for subsequent semi-dense depth estimation.

[0066] (3) This invention provides an improved optimization algorithm for pose estimation and optimization during relative pose transformation of bionic eye cameras. It can effectively optimize the pose of the camera and the relative pose of the left and right bionic eye cameras and the coordinates of landmark points through four threads, providing a foundation for subsequent semi-dense depth estimation steps.

[0067] (4) This invention utilizes the above advantages to perform semi-dense depth estimation, eliminating most interference terms in the depth estimation process and providing good data. At the same time, in order to perform depth estimation more accurately, this invention imposes constraints on the pixel block gradient to eliminate interference terms and optimizes the depth estimation method.

[0068] (5) Compared to dense direct methods, this invention better meets real-time performance requirements for depth estimation. Dense direct methods mainly rely on minimizing the global spatial regularization energy function within a non-convex optimization framework based on keyframes to perform dense depth estimation. This incurs a huge computational burden for each pixel in the image, making it difficult to achieve real-time performance. In contrast, this invention first filters out some image block gradients that do not meet the NCC calculation requirements during block matching. Therefore, compared to dense direct methods, this invention can better meet real-time requirements.

[0069] (6) Compared to using RGB-D cameras for depth estimation, this invention is well-suited for outdoor and large-scale scenes with uneven texture distribution, while reducing power consumption and cost. In large outdoor scenes with uneven texture distribution, RGB-D cameras are susceptible to interference from range, lighting, or infrared light emitted by other sensors. Furthermore, without proper modulation, using multiple RGB-D cameras simultaneously can lead to mutual interference, making it difficult to accurately acquire depth information. Additionally, using multiple RGB-D cameras simultaneously typically incurs higher costs and power consumption. While traditional binocular passive vision has no range limitations, it cannot avoid the impact of uneven texture distribution and dynamic objects on depth estimation. Therefore, this invention utilizes binocular active vision effectively in outdoor and large-scale scenes with uneven texture distribution, while reducing power consumption and cost. Attached Figure Description

[0070] Figure 1This is an overall flowchart of the binocular active vision semi-dense depth estimation method based on a bionic eye in the embodiment;

[0071] Figure 2 This is a flowchart illustrating the use of a panoramic camera to visually guide a binocular bionic eye camera in an embodiment.

[0072] Figure 3 This is a flowchart illustrating how a SLAM system is used to track keyframes and estimate camera pose and landmark coordinates in an embodiment.

[0073] Figure 4 This is a flowchart of the semi-dense depth estimation in the embodiment;

[0074] Figure 5 This is a schematic diagram of triangulation. Detailed Implementation

[0075] The present invention will now be described in detail with reference to specific embodiments.

[0076] As attached Figure 1 As shown, this embodiment provides a more specific binocular active vision semi-dense depth estimation method based on a bionic eye, the steps of which include:

[0077] The S100 utilizes a panoramic camera to visually guide a binocular bionic eye camera, which then acquires high-value images, such as... Figure 2 As shown, the process is as follows:

[0078] S101 uses a panoramic camera to capture information about the surrounding environment. From the panoramic images acquired by the panoramic camera, it detects key information that affects the accuracy and robustness of SLAM operation, including feature points and dynamic objects.

[0079] In panoramic images, to achieve matching with subsequent binocular active SLAM algorithms and ensure that subsequent binocular active SLAM can extract the same feature points, guaranteeing the effectiveness of the panoramic value map constructed from these feature points, and simultaneously achieving real-time performance, the FAST keypoint detection algorithm is used to extract feature points from the panoramic image. Since the panoramic value map only provides information about feature points to control the movement of the binocular bionic eye, the calculation of the BRIRF descriptor in the ORB feature points extracted by the SLAM algorithm is omitted.

[0080] For dynamic objects that severely degrade the performance of the SLAM algorithm, their presence in the input image should be avoided as much as possible. Therefore, detecting dynamic objects in the image is the first step. Existing dynamic object detection algorithms can detect pedestrians, animals, and other potential dynamic objects that negatively impact the SLAM algorithm in panoramic images, thus minimizing interference from dynamic objects.

[0081] S102, evaluate each region in the panoramic image based on key scene information to determine the value of each region to the SLAM algorithm, assign values ​​to pixels in the panoramic image, and construct a panoramic value map:

[0082] Based on the feature points extracted from the panoramic image, each feature point and its surrounding pixels are assigned a high value. The value gradually decreases as the radius of the neighborhood increases. If the same pixel is assigned a value multiple times in an area where feature points are concentrated, the value assigned to that pixel is the maximum value.

[0083] Furthermore, all pixels in the target bounding box for the dynamic object detection described above are assigned a low value, even lower than the value of areas where no feature points are detected, in order to eliminate the impact of dynamic objects on the performance of subsequent SLAM algorithms.

[0084] S103: Project the field of view regions of the left and right bionic eye cameras onto the panoramic value image respectively to obtain the left and right field of view projection regions; simultaneously control the overlap of the left and right field of view projection regions; merge the left and right field of view projection regions to obtain the current binocular field of view projection region:

[0085] Here, the initial pose coordinate system of the panoramic camera relative to the left camera is denoted as T. LOP Let T be the initial pose coordinate system of the panoramic camera relative to the right camera. ROP Furthermore, these two transformation matrices can be determined through calibration. Based on feedback from the bionic eye control module, the left and right bionic eyes, relative to their initial pose T... LOL and T ROR Both can be obtained in real time. The normalized planar projection equations for the two bionic eye cameras are defined as follows: The projection equation from the sphere of the panoramic camera to the pixel plane is: Taking the left camera as an example, the four corner pixels of the left bionic eye camera are u L(0,0) u L(m，0) u L(m，n) and u L(0，n) (where m and n are the maximum pixel coordinates in the width W and height H directions of the bionic eye camera pixel plane, respectively) in its normalized plane P L Projection on:

[0086]

[0087] From this, we can obtain the pixel points at the four corners of the left bionic eye camera in the normalized plane P. L Projection P on L(0,0) P L(m，0) P L(m，n) and P L(0，n) The same applies to the right bionic eye camera.

[0088] Furthermore, the four corner projection points on the normalized plane of the left camera are transformed into the coordinate system of the panoramic camera:

[0089]

[0090] Since the bionic eye camera has only two degrees of freedom—left and right rotation and up and down pitch—its current pose coordinate system coincides with the origin of its initial pose coordinate system. Furthermore, the relative positions of the origins of the panoramic camera and the bionic eye camera remain constant. This can be determined from the T coordinates obtained during calibration. LOP The translation vector t of the left bionic eye camera relative to the panoramic camera is obtained in the middle. PL0 . t PL0 and P L(0，0) P L(m，0) P L(m，n) and P L(0，n) Connect these four points to construct four straight line equations L. P(0，0) L P(m，0) L P(m，n) L P(0，n) The equations of these four lines are expressed in the following forms:

[0091]

[0092] Where, x PL0 Transform the coordinates of the origin of the left bionic eye camera to the x-coordinates in the panoramic camera coordinate system, y-coordinates. PL0 Transform the coordinates of the origin of the left bionic eye camera to the y-axis coordinates in the panoramic camera coordinate system, z-axis coordinates. PL0 The coordinates of the origin of the left bionic eye camera are transformed to the z-direction coordinates in the panoramic camera coordinate system.

[0093] The normalized spherical equation for the spherical projection of a panoramic camera is:

[0094] x 2 +y 2 +z 2 =1 (4),

[0095] By solving equations (3) and (4) simultaneously, we can find the intersection points of the four lines with the panoramic camera's projection sphere. Each line will form two intersection points with the sphere. However, since the bionic eye camera is limited by its range of motion and cannot see scenes where x < 0 in the panoramic camera's coordinate system, we discard the results for x < 0 and retain the intersection points for x > 0. Therefore, the intersection points of the four lines with the panoramic camera's projection sphere are P. P1 P P2 P P3 and P P4 .

[0096] Furthermore, the projection equation from the panoramic camera's projection sphere to the pixel plane is... Project the four intersection points onto the pixel plane of the panoramic camera:

[0097]

[0098] Connecting the four projection points results in an irregular quadrilateral due to distortion during the panoramic camera's projection process. This quadrilateral represents the projection of the left bionic eye camera onto the panoramic image. Similarly, the projection of the right bionic eye camera can be obtained. Based on the mechanical limitations of the bionic eye camera, its range of motion within the panoramic image can be determined. Furthermore, to ensure that all captured images fall within the range of motion of both the left and right bionic eye cameras, their range of motion needs to be further limited. It should be noted that, for the convenience of the panoramic value algorithm, the integrated gaze area projection is approximated as its maximum bounding rectangle in this stage. This simplifies the calculation process by approximating the complex irregular shape as a rectangle, making it easier to handle in subsequent algorithms.

[0099] S104: Obtain the average value of the current binocular field-of-view projection area in the panoramic value image; determine the value of the current binocular field-of-view projection area compared with the value threshold: if the average value of the current binocular field-of-view projection area is greater than the value threshold, then do not move the current binocular field-of-view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system; if the average value of the current binocular field-of-view projection area is less than or equal to the value threshold, then search for a target binocular field-of-view projection area with a higher value than the value threshold in the panoramic value image, and then proceed to step S105.

[0100] By using a rectangle of the same size as the projected gaze area of ​​the current bionic eye camera as a sliding window within the gazeable region of the panoramic image, and searching from near to far in a certain step size s from the current gaze area, combined with the panoramic image value mentioned above, the average value V of each window can be obtained. m To reduce the impact of camera shake during rotation on the SLAM algorithm, a value threshold V is defined here. th If the mean value of the currently viewed region is higher than the value threshold (i.e., V), m >V th If V is within the specified range, it means that the current gaze area meets the requirements for good operation of the SLAM algorithm, and in this case, the camera is not moved. Conversely, if V is not within the specified range, the camera is not moved. m ≤V th This means that the number of extractable feature points in the current scene is insufficient or that the scene contains dynamic objects. In order to improve the running quality of the SLAM algorithm, we choose to shift our focus to areas where the average value is higher than the value threshold V. thAnd the area closest to the current gaze area.

[0101] S105, calculate the required displacement of the left and right bionic eye cameras according to the current binocular field of view projection area and the target binocular field of view projection area; move the left and right bionic eye cameras according to the displacement amount; and finally use the high-value images collected by the left and right bionic eye cameras in the target binocular field of view projection area as the input of the SLAM system.

[0102] As mentioned earlier, the bionic eye has two degrees of freedom: left and right rotation and up and down pitch. It only undergoes pure rotation relative to the robot platform; therefore, the rotational motion of the left bionic eye camera can be represented by the rotation matrix R. LCLT The rotational motion of the right camera is represented by the rotation matrix R. RCRT .

[0103] Taking the left camera as an example, the center pixel coordinates u of the target's gaze area in the panoramic value map are... PT According to the projection model The image is back-projected onto the projection sphere of the panoramic camera, and the projection point P on the sphere is also back-projected. PT Transform to the initial pose coordinate system O of the left camera LO Obtain the target optical axis direction P of the left camera LOT :

[0104]

[0105] Furthermore, the center point of the pixel plane of the area currently being gazed at by the left bionic eye camera... Projecting the model through the camera Back-projected onto the normalized plane, and then transformed into the initial pose coordinate system O of the left bionic eye camera. LO Obtain the left camera in the initial pose coordinate system O LO The current pose P in LOC :

[0106]

[0107] Therefore, the left bionic eye camera needs to be adjusted from the current pose P LOC Move to target pose P LOT .

[0108] Furthermore, the left bionic eye camera calculates from the current pose P LOC Move to target pose P LOT The rotation matrix R between LCLT First, calculate the rotation vector between the two vectors representing the attitude.

[0109] Determine the axis of rotation between two vectors:

[0110]

[0111] Since the axis of rotation in the rotation vector is a unit vector, the vector after the cross product in the above equation is normalized. According to the definition of the dot product, for a rotation angle θ... LCLT :

[0112]

[0113] The rotation matrix R can be obtained using the Rodriguez formula based on the rotation angle and rotation axis. LCLT :

[0114]

[0115] By solving the rotation matrix R of the left camera LCLT This allows us to obtain the rotation commands that the bionic eye control module needs to issue to the left bionic eye control motor, enabling the bionic eye camera to shift its field of view to a high-value area in the scene, thus achieving active vision. Similarly, the rotation matrix of the right camera can be obtained, and its rotation can be controlled to improve the performance of the SLAM algorithm.

[0116] S200 inputs the high-value images acquired by S100 into the SLAM algorithm, which then estimates the camera pose and landmark coordinates.

[0117] like Figure 3 As shown, this embodiment uses an improved SLAM algorithm based on the ORB-SLAM2 framework to ensure stable operation when the left and right bionic eye cameras move relative to each other, and to achieve higher accuracy in camera pose and landmark coordinate estimation.

[0118] The SLAM algorithm consists of four modules: a tracking thread, a local mapping thread, a loop closure thread, and a global basis analysis (BA) thread. Specifically:

[0119] S201, the tracking thread, estimates the camera pose and determines keyframes:

[0120] First, an initial local map is constructed: images acquired by the left and right bionic eye cameras are input into the SLAM algorithm framework. ORB feature points from both images are extracted in the preprocessing module. All subsequent algorithm modules operate based on these extracted ORB feature points. The image representation is transformed from a set of pixels to a set of ORB feature points, reducing the amount of data cached during runtime. The pose T between the left and right bionic eyes is obtained based on calibration. L0R0 Furthermore, an initial local map is constructed using binocular parallax.

[0121] Secondly, the left camera is designated as the main camera. The preprocessed image input from the left camera is processed using ORB feature point extraction. The left camera pose is initialized using motion mode, PnP relocalization mode, or reference keyframe mode. At this point, the initial pose estimation accuracy is low, requiring further optimization by tracking the local map. ORB feature points matching the current left camera frame are retrieved from the local map, and then a PnP problem is constructed to minimize the reprojection error, thereby optimizing the left camera pose.

[0122] For the error term defined in nonlinear optimization:

[0123]

[0124] Where u i This represents the pixel coordinates of the observed point P. i Let P represent the 3D coordinates of point P in the local map; e represents the reprojection error; taking camera pose as an example, when constructing the least squares problem, the optimal camera pose needs to be obtained:

[0125]

[0126] This least squares problem can be constructed into an unconstrained optimization problem using Lie algebras, which can be easily solved by optimization algorithms. During iterative optimization, the derivative of the reprojection error with respect to the camera pose is used, which is the Jacobian matrix required to linearize the objective function.

[0127]

[0128] The update amount of the left camera pose can be calculated.

[0129] Furthermore, the pose estimation task of the tracking thread also needs to estimate the relative poses of the left and right cameras. The poses T of the left and right bionic eyes relative to their initial pose coordinate system have already been obtained during the aforementioned bionic eye motion control process. LOL and T ROR This allows us to calculate the initial relative pose T between the left and right bionic eyes. LR :

[0130]

[0131] After the left camera completes pose estimation, it is successfully matched with landmarks in the local map. Then, based on the ORB feature point matching between the left and right cameras, and the matching of ORB feature points in the left camera with landmarks, a PnP problem is constructed to minimize the feature point reprojection error between the left and right cameras, resulting in an optimized relative pose T between the left and right cameras. LR Thus, the current pose of the left camera and the relative pose of the left and right cameras have been initially optimized; at this point, the optimized relative pose transformation matrix T of the left and right cameras is used.LR In addition to the well-matched feature points already obtained in the image, triangulation can be performed to obtain a large number of high-quality binocular matching landmarks. These high-quality landmarks, as a supplement to the landmarks in the local map, improve the accuracy and robustness of the tracking thread.

[0132] Finally, keyframes are selected to serve subsequent threads. Here, a keyframe refers to all feature points in the left camera image within a frame that meets the requirements. The image data corresponding to the keyframe does not participate in the SLAM algorithm but will enter the semi-dense depth estimation module. The current frame is set as a keyframe if it meets any of the following conditions:

[0133] (1) The current frame is the first frame after the bionic eye gaze control reaches the target gaze area;

[0134] (2) More than 13 frames have passed since the last global relocation;

[0135] (3) The local mapping thread is in an idle state;

[0136] (4) More than 15 frames have passed since the last keyframe set;

[0137] (5) The translation distance between the current frame and the previously set keyframe exceeds the threshold t. th ;

[0138] (6) The number of successfully tracked feature points in the current frame reaches 70 or more;

[0139] (7) The number of successfully tracked feature points in the current frame is less than 85% of that in the reference keyframe.

[0140] The selection of keyframes provides a prerequisite for subsequent threads.

[0141] S202, the local mapping thread, manages the local map and performs local BA optimization. Its purpose is to update and remove keyframes and landmarks in the local map, while optimizing the camera pose of the keyframes and the coordinates of the landmarks.

[0142] First, keyframes are updated. In the tracking thread, once it is determined that the current frame meets the conditions for being set as a keyframe, this new keyframe is associated with the previous keyframes. This association process is based on the co-visibility relationship of landmarks between the new keyframe and the previous keyframes, and the bag-of-words representation of the new keyframe is calculated based on the DBoW bag-of-words library.

[0143] Simultaneously, landmarks are updated and maintained; if a landmark is to be retained in the local map, it must meet two conditions in the first three keyframes after its creation:

[0144] (1) The landmark must be successfully tracked in more than 25% of the frames in which the landmark is visible based on the pose prediction;

[0145] (2) If more than one keyframe is added after the landmark is created, the landmark must be observed by at least three keyframes.

[0146] In the local mapping thread, local BA optimization essentially incorporates the coordinates of landmark points as parameters into the optimization. Similarly, in the tracking thread, minimizing the reprojection error in the PnP operation optimizes the camera pose. Using the derivative of the reprojection error with respect to the landmark coordinates, and following the chain rule, we have:

[0147]

[0148] The update amount of landmark coordinates can be obtained. When the objective function error converges, the camera pose and landmark coordinates of the optimal keyframe in the local map can be obtained.

[0149] Furthermore, to prevent the data size in the local BA from gradually increasing and affecting the real-time performance of the local mapping thread, keyframes are removed. When 90% of the landmarks in a keyframe can be observed in at least three other keyframes, the keyframe is removed, but its image data and pose will still enter the semi-dense depth estimation module.

[0150] S203, Loop Closure Thread: This thread uses the DBoW2 library to detect large loop closures and corrects accumulated camera trajectory errors by performing pose graph optimization. Its main task is to eliminate the accumulated errors caused by the tracking and local mapping threads, which only consider adjacent timeframes.

[0151] First, in the loop closure detection phase, the system detects possible loop closures and identifies candidate loop closure keyframes. Next, it calculates the relative pose between the current keyframe and these candidate loop closure keyframes. Using their co-location relationships, the loops are closed. Then, by using the calculated relative poses between the current keyframe and the loop closure keyframes, the loops are corrected through pose graph optimization. Pose graph optimization only considers the trajectory, not landmark optimization; that is, it constructs a graph optimization that only considers the trajectory. Specifically, assuming there are K... i K j The two keyframes, with camera poses T before loop closure correction, are respectively... Wi and T Wj After correcting the camera pose during loop closure detection, the relative pose T between two frames can be obtained. ij Therefore, the error e is constructed. ij :

[0152]

[0153] Based on this error, a nonlinear optimization is constructed, and the error terms are solved with respect to T. Wi and T Wj The derivative of the expression can be used for linearization, thereby optimizing the pose graph.

[0154] S204, Global BA Thread. After completing pose graph optimization, the global BA thread is started. Its main task is to optimize the final accurate trajectory and map using BA after no new data input is available, obtaining globally consistent camera pose and map structure.

[0155] After the loop closure thread completes the pose graph optimization, a global optimization is performed to obtain the globally optimal solution throughout the process. After optimization, the updated keyframe camera poses are merged with the unupdated keyframe camera poses using a spanning tree, and the coordinates of landmark points are corrected based on the updated keyframe camera poses of their reference points. If a new loop is detected during the global BA optimization process, the thread is terminated and restarted after the pose graph optimization is completed.

[0156] S300 inputs the optimized keyframe pose and its corresponding keyframe image data obtained in S200 into the semi-dense depth estimation module. The semi-dense depth estimation module uses epipolar search, block matching combined with triangulation, and then utilizes a depth filter to achieve semi-dense depth estimation. Figure 4 As shown, the construction of a semi-dense map is then achieved, and the steps include:

[0157] S301, Select one frame in the semi-dense depth estimation module as the reference keyframe, and assume that the inverse depth corresponding to all pixel depths in the image data corresponding to the reference keyframe satisfies an initial Gaussian distribution. The purpose is to provide an initial value for subsequent depth information fusion.

[0158] Taking the image data corresponding to one of the reference keyframes as an example, let the inverse depth value of a certain pixel be d. -1 And it follows a Gaussian distribution:

[0159] P(d -1 )=N(μ,σ 2 (17),

[0160] The initial mean and variance of the inverse depth distribution have a certain impact on the convergence speed.

[0161] S302, when the optimized new keyframe pose and its corresponding image data enter the semi-dense depth estimation module, the optimized keyframe pose is used to perform epipolar search, and further block matching is performed to determine the projection point position.

[0162] Let u1 be a pixel in the image corresponding to the reference keyframe, and let l2 be the epipolar line projecting the depth range corresponding to this pixel into the image corresponding to the current keyframe. Let I denote the image corresponding to the reference keyframe. k The image corresponding to the current keyframe is denoted as I. t The rotation matrices of the world coordinate system relative to the reference keyframe and the current keyframe are R and R, respectively. kW and R tW The translation vectors of the world coordinate system relative to the reference keyframe and the current keyframe are t and t, respectively. kW and t tW This can be obtained using the SLAM algorithm described above. Simultaneously, the intrinsic parameters K of the left camera can be obtained through the aforementioned calibration. L The relationship between the reference keyframe and the current keyframe in pixel coordinates is as follows:

[0163]

[0164] Among them, u k To reference a pixel in a keyframe, d k This represents the depth of the spatial point corresponding to that pixel. When d is known... k u k When u is available, it can be calculated. t Position; when given u k When two components are further increased by an increment du and dv, u can be calculated. t Incremental du t dv t Therefore, an affine transformation consisting of a linear relationship between the coordinate transformations of the keyframe and the current frame within a local range can be obtained:

[0165]

[0166] Next, based on the affine transformation matrix, the camera rotation is considered in block matching. In feature matching methods, it's necessary to calculate the matching between features based on descriptors; however, it's not possible to calculate a descriptor for every pixel. Therefore, a comparison of brightness values ​​is needed. Since the brightness of a single pixel is almost indistinguishable, it's necessary to compare the brightness of pixel blocks, i.e., using block matching.

[0167] Take the pixel block surrounding u1 as At the same time, the pixel block on l2 is recorded as The mean-removed normalized cross-correlation method is used to compare the correlation between the pixel blocks surrounding u1 and the pixel blocks on the epipolar line. The mean-removed normalized cross-correlation method allows for some changes in the overall brightness of the pixel blocks, and is more reliable than the general normalized cross-correlation method.

[0168] When the angle between the pixel block gradient and the epipolar line is large, the reliability of block matching is low. Therefore, only pixel blocks whose pixel gradient is in the epipolar direction or within a certain threshold in the epipolar direction are matched. According to the NCC definition, we have:

[0169]

[0170] After determining the step size, a non-convex NCC distribution function can be obtained by traversing the NCC values ​​of the specified pixel blocks and the source pixel blocks along the corresponding epipolar lines. The point with the highest NCC is found as the matching pixel block of the source pixel block; if the highest matching value is also lower than the threshold, the epipolar line search and block matching for the source pixel block in the image corresponding to the current keyframe are considered to have failed, and new image and pose data are needed before matching and information fusion convergence can be performed.

[0171] The restriction on pixel block gradients effectively reduces the computation of mean-normalized cross-correlation during the matching process and reduces mismatches of pixel blocks, thus obtaining a more accurate triangulated depth. Without this restriction, the computational load would increase, and the uncertainty of block matching would also increase, resulting in ineffective matching and potentially leading to incorrect depth values, which in turn would affect the subsequent information fusion process.

[0172] Furthermore, the estimated depths in the reference keyframe obtained by block matching and triangulation of images corresponding to different current keyframes and reference keyframes may differ, while the coordinates of a point in 3D space are fixed. Therefore, a depth filter is needed to describe the depth value through a probability distribution.

[0173] S303 calculates the triangulated 3D point depth and its uncertainty based on the geometric relationship between the reference keyframe and the current keyframe. Its main task is to provide a Gaussian distribution of the inverse depth corresponding to the new observation depth so that subsequent information fusion can bring the depth data together.

[0174] The calculation of the depth of a 3D point uses triangulation (triangulation measurement) method, as shown in the diagram below. Figure 5 As shown, the calculation method is as follows:

[0175] Suppose we have two image frames, I1 and I2, with I1 as the reference and the transformation matrix of I2 as T. The camera optical centers are O1 and O2. There is a feature point p1 in I1, and a corresponding feature point p2 in I2. Connecting the optical center O1 to feature point p1, and the optical center O2 to feature point p2, we obtain rays O1p1 and O2p2. Theoretically, when the camera does not perform pure translation, lines O1p1 and O2p2 should intersect at a point P in the scene. This point represents the location of the map points corresponding to the two feature points p1 and p2 in the 3D scene. Due to noise, these two lines may not intersect, which can be solved using the least squares method.

[0176] Let x1 and x2 be the normalized coordinates corresponding to feature points p1 and p2, then their relationship satisfies:

[0177] s2x2=s1Rx1+t (21),

[0178] Where s1 and s2 are the depths of two feature points p1 and p2, respectively, and the rotation matrix R and translation vector t are known. Geometrically, a 3D point can be found on ray O1p1 such that its projected position is close to p2. Similarly, it can be found on ray O2p2. Taking the calculation of s1 as an example, transforming the above equation yields an equation only concerning s1:

[0179]

[0180] Solving the above equations directly yields s1. Similarly, s2 can also be calculated. At this point, the depths of the two feature points can be preliminarily determined. Due to the presence of noise, a least-squares solution is required.

[0181] As new keyframes are continuously generated, epipolar search and block matching are used to find corresponding pixel blocks, and triangulation is used to estimate depth. The observed depth in the reference keyframe is calculated and converted into inverse depth. This inverse depth also follows a Gaussian distribution.

[0182]

[0183] in, The depth deviation can be obtained by combining the pixel error after epipolar search with the geometric relationship of triangulation.

[0184] S304: The Gaussian distribution of the inverse depth corresponding to the current observation depth is fused into the previous estimate to establish the fused inverse depth distribution. If the uncertainty is less than a certain threshold, the depth data is considered to have converged; otherwise, information fusion needs to continue. Steps S303 and S304 together constitute a depth filter.

[0185] First, by fusing the Gaussian distributions of the old and new inverse depths, the product of the Gaussian distributions yields:

[0186]

[0187] By iterating through the above inverse depth observation calculation and fusion process, when the uncertainty obtained from the latest information fusion is less than a certain threshold, the depth data can be considered to have converged.

[0188] Further processing, such as drawing and reading data, can eventually yield a semi-dense depth map.

[0189] This invention, based on the visual mechanism of the human eye, constructs a panoramic value map in scenes with uneven feature point distribution, assigning higher value to areas with dense feature points. This strategy guides the bionic eye camera towards these areas. Dense feature points facilitate pose estimation optimization and global map construction, thereby enhancing the stability of the SLAM system. Simultaneously, feature-rich areas provide a texture foundation for subsequent block matching in semi-dense depth estimation, reducing the likelihood of mismatches in areas with clear textures. Secondly, the panoramic value map construction eliminates interference from dynamic objects in semi-dense depth estimation. If dynamic objects appear during semi-dense depth estimation, they affect the block matching process. After a dynamic object moves, pixels that should have been matched on the corresponding epipolar line disappear, leading to incorrect pixel block matching and resulting in matching failure or depth estimation errors. Therefore, eliminating dynamic objects provides higher-quality image data for subsequent semi-dense depth estimation.

[0190] Meanwhile, the SLAM algorithm for relative pose transformation of the bionic eye camera was improved for binocular active vision, laying the foundation for subsequent semi-dense depth estimation.

[0191] In the semi-dense depth estimation module, compared to traditional passive vision binocular semi-dense depth estimation, where the baseline between the cameras is fixed and pixel depth can be calculated using the parallax of the left and right cameras, the relative poses between the binocular bionic eye cameras change in this invention. Therefore, the traditional parallax-based method fails. However, this invention uses an improved SLAM algorithm to obtain the selected keyframe poses. Therefore, using the keyframe poses and their corresponding image data, epipolar search and block matching are used to find the corresponding pixel blocks, and triangulation is used to estimate the depth. Certain restrictions are placed on the direction of the pixel block gradient during block matching, resulting in more accurate triangulated depth. Furthermore, binocular active vision provides more texture information than traditional monocular passive semi-dense depth estimation, further improving depth estimation accuracy. This invention has significant application value in real-world scenarios, providing an effective method for robots to autonomously navigate and locate in unknown environments.

Claims

1. A binocular active vision semi-dense depth estimation method based on a biomimetic eye, characterized in that, The method comprises the following steps: S100, acquiring images by using binocular bionic eye cameras; In the step S100, a panoramic camera is used to guide the gaze of the binocular bionic eye cameras, and the step comprises: S101, acquiring a panoramic image, and then detecting key scene information in the panoramic image, including feature points and dynamic objects, which affect the execution of a SLAM task; S102, assigning values to pixel points in the panoramic image based on the key scene information, and constructing a panoramic value image; S103, projecting the field of view regions of the left and right bionic eye cameras into the panoramic value image respectively to obtain a left field of view projection region and a right field of view projection region; the left field of view projection region and the right field of view projection region are controlled to overlap; and the left field of view projection region and the right field of view projection region are merged to obtain a current binocular field of view projection region; S104, obtaining a value mean of the current binocular field of view projection region in the panoramic value image; judging the size of the value mean of the current binocular field of view projection region and a value threshold: if the value mean of the current binocular field of view projection region is greater than the value threshold, the current binocular field of view projection region is not moved, and high-value images currently acquired by the left and right bionic eye cameras are taken as inputs of a SLAM system; if the value mean of the current binocular field of view projection region is less than or equal to the value threshold, a target binocular field of view projection region with a value higher than the value threshold is searched in the panoramic value image, and then the step S105 is entered; S105, calculating displacement amounts required by the left and right bionic eye cameras respectively according to the current binocular field of view projection region and the target binocular field of view projection region; moving the left and right bionic eye cameras respectively according to the displacement amounts, and finally taking high-value images acquired by the left and right bionic eye cameras in the target binocular field of view projection region as inputs of the SLAM system; S200, inputting images acquired in the step S100 into the SLAM system, tracking key frames and estimating camera poses and landmark point coordinates by using the SLAM system; S300, using key frame poses and corresponding key frame image data obtained in the step S200, realizing semi-dense depth estimation by using a depth filter after polar search, block matching and triangulation, and the specific process is as follows: S301, selecting a frame entering the step S300 as a reference key frame, assuming inverse depths corresponding to depths of all pixels in image data corresponding to the reference key frame as satisfying an initial Gaussian distribution, and providing an initial value for subsequent depth information fusion; S302, inserting a new key frame, and determining projection point positions by using polar search and block matching; S303, calculating depths and uncertainties of three-dimensional points after triangulation according to geometric relations between the reference key frame and the current key frame; S304, fusing Gaussian distributions of inverse depths corresponding to current observation depths into distributions of inverse depths after fusion by using the last estimation; if the uncertainty is less than a certain threshold, it is considered that the depth data converges, the depth of the three-dimensional point is determined, otherwise, the step S302 is returned to continue information fusion.

2. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 1, wherein, The step S302 is specifically as follows: Let a pixel in the image corresponding to the reference key frame be u1, the projection of the depth range corresponding to the pixel in the image corresponding to the current key frame be an extreme line l2; the image corresponding to the reference key frame is denoted as I k , the image corresponding to the current key frame is denoted as I t , the rotation matrix of the world coordinate system relative to the reference key frame and the current key frame is R kW and R tW , the translation vector of the world coordinate system relative to the reference key frame and the current key frame is t kW and t tW , the intrinsic parameter K L of the left camera is obtained through calibration, and the relationship between the images corresponding to the reference key frame and the current key frame in the pixel coordinate system is as follows: wherein u k is a pixel in the reference key frame, d k is the depth of the spatial point corresponding to the pixel; when d k , u k is known, the position of u t can be calculated; when the two components of u k are increased by an increment du, dv, the increment du t , dv t of u t can be calculated, thus obtaining an affine transformation constituted by a linear relationship of the image coordinate transformation between the key frame and the current frame in a local range: Then, the rotation of the camera is considered in the block matching according to the affine transformation matrix: Let the pixel block around u1 be Meanwhile, let the pixel block on l2 be The normalized cross-correlation method is used to compare the correlation of the pixel block around u1 and the pixel block on the epipolar line. When the angle between the pixel block gradient and the angle between the epipolar line is large, the reliability of the block matching is low, so only the pixel block whose gradient is in the direction of the epipolar line or within a certain threshold of the epipolar line is matched. According to the definition of NCC, there is: After the step length is determined, a non-convex NCC distribution function can be obtained by traversing the NCC values of the prescribed pixel block and the source pixel block on the epipolar line, and the point with the highest NCC is taken as the matching pixel block of the source pixel block.

3. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 2, wherein, In step S302, if the highest NCC value obtained by block matching is still lower than the threshold value, it is considered that the epipolar search and block matching for the source pixel block in the image corresponding to the current key frame fail, and the matching and information fusion convergence are performed again after entering a new key frame and the camera pose corresponding to the new key frame and optimized by the SLAM system in S200.

4. The biomimetic eye based binocular active vision semi-dense depth estimation method of claim 1, wherein, In step S303, the depth of the three-dimensional point is calculated by triangulation, and the calculation method is as follows: Suppose there are two images I1 and I2, I1 is taken as the reference, and the transformation matrix of I2 is T; the camera optical centers are O1 and O2; there is a feature point p1 in I1, and a feature point p2 in I2; the rays O1p1 and O2p2 are obtained by connecting the optical center O1 and the feature point p1, and the optical center O2 and the feature point p2; when the camera does not make pure translation motion, the straight lines O1p1 and O2p2 intersect at a point P in the scene, which is the position of the map point corresponding to the two feature points p1 and p2 in the three-dimensional scene; Suppose x1 and x2 are the normalized coordinates of the feature points p1 and p2, then their relationship satisfies: s2x2=s1Rx1+t, wherein s1 and s2 are the depths of the two feature points p1 and p2, the rotation matrix R and the translation vector t are known; From the geometric point of view, the three-dimensional point can be found on the ray O1p1 to make its projection position close to p2, and the same is true for the ray O2p2. Taking s1 as an example, the above equation is transformed to get an equation only about s1: The equation is solved to directly obtain s1; s2 is obtained in the same way; at this time, the depths of the two feature points are preliminarily determined; When O1p1 and O2p2 cannot intersect due to noise, the least square method is used to solve.

5. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 4, wherein, When new current key frames are continuously generated, the corresponding pixel blocks are found by epipolar search and block matching, and the depth is estimated by triangulation, the observation depth in the reference key frame is calculated and converted into inverse depth, and the inverse depth is also a Gaussian distribution at this time. wherein, The depth bias is obtained by combining the error of the epipolar search with the depth calculated from the triangulation geometry.

6. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 5, wherein, In step S104, By using the same size rectangular as the current biomimetic eye camera gaze region projection in the panoramic image gazeable region as a sliding window, search from the current gaze region to the far with a certain step s, and combine the panoramic image value, get the value mean V in each window m ; defining a value threshold V th If the average value of the current gaze region is higher than the value threshold, the camera is not moved. Otherwise, if the average value of the current gaze region is lower than or equal to the value threshold V, the gaze is shifted to the region having the highest value and being the closest to the current gaze region. th If the average value of the current gaze region is higher than the value threshold, the camera is not moved. Otherwise, if the average value of the current gaze region is lower than or equal to the value threshold V, the gaze is shifted to the region having the highest value and being the closest to the current gaze region.

7. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 5, wherein, In the step S105, the rotational motion of the left bio- eye camera is represented as a rotation matrix R LCLT , the rotational motion of the right camera is represented as a rotation matrix R RCRT , and R LCLT is solved by the following method: The center pixel coordinate u of the target gaze region in the panoramic value map PT According to the projection model Back-project to the panoramic camera projection sphere, and convert the back-projected point P to the sphere to the point P PT Convert to the initial pose coordinate system O of the left camera LO Obtain the target optical axis direction P of the left camera in the initial pose coordinate system O LOT : Convert the center point of the pixel plane of the current gaze region of the left biomorphic eye camera to the initial pose coordinate system O of the left camera Through the camera projection model Back-project to the normalized plane, and convert it to the initial pose coordinate system O of the left biomorphic eye camera LO Obtain the current pose P of the left camera in the initial pose coordinate system O LO : LOC : Therefore, the left biomorphic eye camera needs to move from the current pose P to the target pose P LOC : LOT ; Compute current pose P LOC Rotation axis between target pose P LOT and current pose P According to the definition of dot product, for the rotation angle θ LCLT : According to the rotation angle and the rotation axis, the rotation matrix R can be obtained by the Rodrigues formula LCLT : Solve for R as well RCRT ; By solving the rotation matrix of the left and right cameras, the rotation instructions required by the bionic eye control module for the left and right bionic eye control motors are obtained, so that the bionic eye camera shifts the field of view to the high-value area in the scene, and the gaze guidance is realized.

8. The biomimetic eye based binocular active vision semi-dense depth estimation method according to any one of claims 1-7, characterized in that, The SLAM system adopts a SLAM algorithm improved based on the ORB-SLAM2 framework, which includes four modules, namely, a tracking thread, a local mapping thread, a loop closure thread, and a global BA thread. (1) The tracking thread is realized by matching the feature points of the current frame and the local map, the camera pose is initialized and tracked by using the motion mode, the PnP repositioning mode or the reference key frame mode, the pose of the camera is estimated by minimizing the re-projection error, and the key frame is determined; (2) The local map is managed and the local BA optimization is performed by the local mapping thread; (3) The loop closure thread is formed by detecting large loops by using the DBoW2 library and correcting the cumulative error of the camera trajectory by performing pose graph optimization; (4) After the pose graph optimization is completed, a global BA thread is started to obtain globally consistent camera poses and map structure.

9. The biomimetic eye-based binocular active vision semi-dense depth estimation method of claim 8, wherein, The SLAM system finally sets the current frame as a key frame if the current frame meets any of the following conditions: (1) The current frame is the first frame after the biomimetic eye line-of-sight control reaches the target gaze region; (2) The distance from the last global relocalization is more than 13 frames; (3) The local mapping thread is in an idle state; (4) The distance from the last set key frame is more than 15 frames; (5) the translation distance between the current frame and the last set key frame exceeds a threshold t th ; (6) The number of successfully tracked feature points in the current frame is more than 70; (7) The number of successfully tracked feature points in the current frame is less than 85% of the reference key frame.

Images