Self-supervised image depth estimation method based on channel self-attention mechanism

Abstract

The application provides a self-supervised image depth estimation method based on a channel self-attention mechanism, compared with the prior art, the self-supervised depth estimation network based on the channel attention mechanism constructed in the application can reduce the optimization difficulty of the network by using the sparse landmark points obtained by the SLAM system for network pre-training, and can relieve the problem of slow convergence; meanwhile, in the training process of the depth estimation network, the estimation task of the camera pose is handed over to the SLAM system, instead of using the pose estimation network to estimate the camera pose, and the camera pose obtained by the SLAM is used for reconstructing the target image, so that the joint optimization of the depth estimation network and the pose estimation network can be avoided, the burden of network training can be reduced, and the difficulty of network learning can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method for depth estimation using a self-supervised depth estimation network based on a channel self-attention mechanism, and further constructing a dense map. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) uses sensor input data to calculate the relative pose (position and attitude) transformation of the robot in real time to realize the localization of the robot, and simultaneously incrementally constructs the map of the scene during the movement of the robot. According to the used sensors, it can be divided into visual SLAM and laser SLAM. Visual SLAM has the advantages of low cost and rich image information, and therefore performs well in environments with similar geometric mechanisms and in dealing with loop detection problems. However, when a robot with a visual sensor runs SLAM in an unknown environment, the system basically passively receives image data collected by the visual sensor during the movement of the robot, and perceives the pose of the robot and the surrounding environment according to these data. When there are few textures and many dynamic objects in the images collected by the visual sensor in the task, the SLAM system will face the problem of tracking failure. Therefore, based on the inspiration of human peripheral vision and central vision perception mechanism, a bionic eye system with peripheral vision and central vision can be constructed, so that the system has active vision capability. When performing the SLAM task, the bionic eye can actively acquire key information in the visual information of the whole scene, and control the movement to shift the line of sight to the key information (high value) area.

[0003] Depth estimation is a subtask in the field of computer vision, and its purpose is to obtain the distance between the object and the shooting point, providing depth information for a series of tasks such as three-dimensional reconstruction, distance perception, SLAM, visual odometry, live detection, video interpolation, image reconstruction, etc. With the rapid development of deep neural networks, monocular or binocular depth estimation based on deep learning has been widely studied and good accuracy has been achieved. For example, a deep neural network is used to perform end-to-end dense depth map estimation on a single image. In order to improve the accuracy of depth estimation, researchers have proposed different network structures, loss functions and training strategies.

[0004] In order to further reduce the network training burden and reduce the network learning difficulty, the present application constructs a depth estimation network based on a channel self-attention mechanism, and uses the camera pose obtained by the SLAM system for reconstruction of the target image, which can avoid the joint optimization of the depth estimation network and the pose estimation network, and only needs to optimize one depth estimation network; and uses the sparse landmark points obtained by the SLAM system for network pre-training, thereby reducing the optimization difficulty of the network and alleviating the problem of slow convergence. SUMMARY

[0005] The application aims to provide a self-supervised deep estimation network based on channel self-attention mechanism, and a method for image depth estimation.

[0006] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0007] The self-supervised image depth estimation method based on the channel self-attention mechanism comprises the following steps:

[0008] Images are collected by using binocular biomimetic eye cameras;

[0009] One of the biomimetic eye cameras is defined as a main camera, a target image is defined as a main camera image, and a source image is defined as two frames of main camera images and another biomimetic eye camera image adjacent to the target image before and after;

[0010] The images collected by the left and right biomimetic eye cameras are input into a SLAM system, and the camera pose and the sparse depth map of the target image are predicted by using the SLAM system;

[0011] The target image and the sparse depth map obtained by the SLAM thread are input into a depth estimation network based on a channel self-attention mechanism architecture to obtain a scene depth map of the target image, wherein the depth estimation network based on the channel self-attention mechanism architecture comprises an encoder, a structure perception module and a decoder, and the parameters of the depth estimation network based on the channel self-attention mechanism architecture are updated in the following manner:

[0012] The target image is input into the encoder, the encoder uses a ResNet-18 network as a backbone to extract semantic features, then the semantic features are input into the structure perception module to generate new features, and the perception of the scene structure is explicitly enhanced; the new feature map generated by the structure perception module is input into the decoder, the decoder first performs 3*3 convolution and up-sampling on the new feature map generated by the structure perception module, then enters the detail perception module, and the feature map obtained by the detail perception module is subjected to 2 times of 1*1 convolution and sigmoid function calculation; a dense depth map of the original resolution is obtained after decoding;

[0013] The sparse depth map obtained by the SLAM system is input into the last network structure of the decoder to participate in the pre-training of the depth estimation network;

[0014] Based on the relative pose between the target image frame and the adjacent frame or another camera in the binocular camera, the target image dense depth map predicted by the channel self-attention mechanism based depth estimation network is projected and reconstructed on the target image according to the relative pose between the frames, and then the re-projection error is constructed between the target image and the reconstructed target image, and the re-projection error is minimized during training.

[0015] Further, the operation process of the structure perception module includes:

[0016] S101, given the feature map generated by the ResNet-18 encoder Firstly, F is reshaped to Where N = H x W is the number of pixels, then F is multiplied by the transpose matrix of F to calculate the feature similarity

[0017] Where i, j represent any two channels, S ij represents the feature similarity of the two channels;

[0018] S102, convert the similarity S to the discriminant by element-wise subtraction

[0019] D ij = max i (S)-S i,j , D ij represents the influence of the jth channel on the ith channel;

[0020] S103, apply the softmax layer to obtain the attention map

[0021] A ij represents focusing attention on a specific part of the two channels, extracting key information in the channel while ignoring irrelevant information;

[0022] S104, multiply the attention map A and the transpose matrix of F, and reshape the result to Let F and Element-wise sum operation between the results to obtain the final output

[0023]

[0024] The detail perception module restores the resolution by fusing the high-level feature H and the low-level feature L of the skip connection, and the specific operation process includes:

[0025] S201 first concatenates the low-level feature L and the high-level feature H, then uses a convolutional layer and performs batch normalization to obtain U, in order to balance the scale of the features:

[0026] Where f() represents a connection, This represents a 3×3 or 1×1 convolution, BN represents batch normalization, and ReLU is used as the activation function σ().

[0027] S202 compresses U into a vector using global average pooling to obtain the global context, and uses two 1×1 convolutional layers and a sigmoid function to compute the weight vector. To recalibrate channel characteristics while simultaneously measuring their importance:

[0028] Where H and W are the height and width of U, and δ() represents the sigmoid function;

[0029] S203, perform element-wise multiplication between V and U to generate a reweighted feature, then sum the reweighted feature with U to obtain the final output O:

[0030] O = V⊙U + U, where ⊙ is the element-wise dot product.

[0031] In the image depth estimation method, the left camera in the binocular bionic eye camera is defined as the main camera, and the target image I... t The source image is two adjacent frames of the left bionic eye camera image I, representing the images from the left bionic eye camera. s ∈{I t-1 ,I t+1} and right bionic eye camera image I s =I tr The pose of the target image relative to the source image is T. st The pose was obtained by the SLAM system;

[0032] A photometric reprojection loss function L is constructed based on reprojection error. p Minimize L p This updates the weight parameters of the depth estimation network based on the channel self-attention mechanism, trains the depth estimation network, and uses the photometric reprojection loss function L. p The definition of is:

[0033] Where pe() represents the photometric error composed of L1 distance and structural similarity SSIM, I s→t The image represents the reconstructed image;

[0034]

[0035] I s→t = s (proj(D t , T st , K)),

[0036] where proj() denotes the predicted depth map D t in I t , α = 0.85, <·> represents the bilinear interpolation sampling operation, K represents the camera intrinsic, D t represents the predicted depth map, T st represents the camera pose obtained by the SLAM system;

[0037] where μ x , μ y represents the luminance mean, represents the brightness index in SSIM, σ x , σ y represents the luminance standard deviation, represents the contrast index, σ xy is the luminance covariance, and the ratio of the product σ x σ y to the standard deviation represents the structure index, C1 and C2 are constants to avoid errors caused by the denominator being 0.

[0038] During the training of the depth estimation network based on the channel self-attention mechanism, an edge-aware smoothing loss L s is also defined:

[0039] where, is the mean normalized inverse depth;

[0040] The luminance reprojection loss L p is combined with the edge-aware smoothing loss L s as the final training loss function of the depth estimation network based on the channel self-attention mechanism:

[0041] where S is the scale number, and λ is the weight of the edge-aware smoothing loss term.

[0042] In the image depth estimation method, the panoramic camera is used to guide the gaze of the binocular biomimetic eye camera, and high-value images are collected as the input of the SLAM system:

[0043] S301, acquire panoramic images through the panoramic camera, and detect key scene information in the panoramic images that affect the execution of the SLAM task;

[0044] S302, value the pixel points in the panoramic images based on the key scene information to obtain a panoramic value image;

[0045] S303, project the field of view regions of the left and right two biomimetic eye cameras into the panoramic value image respectively to obtain a left field of view projection region and a right field of view projection region; meanwhile, control the left field of view projection region and the right field of view projection region to overlap; and merge the left field of view projection region and the right field of view projection region to obtain a current binocular field of view projection region;

[0046] S304, obtain a value mean of the current binocular field of view projection region in the panoramic value image;

[0047] S305, judge the size of the value mean of the current binocular field of view projection region and a value threshold: if the value mean of the current binocular field of view projection region is greater than the value threshold, do not move the current binocular field of view projection region, and take the high value image currently collected by the left and right two biomimetic eye cameras as the input of the SLAM system; if the value mean of the current binocular field of view projection region is less than or equal to the value threshold, search for a target binocular field of view projection region with a value higher than the value threshold in the panoramic value image, and then enter step S306;

[0048] S306, calculate the displacement amount required by the left and right two biomimetic eye cameras according to the current binocular field of view projection region and the target binocular field of view projection region respectively; move the left and right two biomimetic eye cameras according to the displacement amount respectively, and finally take the high value image collected by the left and right two biomimetic eye cameras in the target binocular field of view projection region as the input of the SLAM system.

[0049] The key scene information includes feature points and dynamic objects; the feature points are extracted by using a FAST key point detection algorithm; and the detection process of the dynamic objects is detecting the dynamic objects by using a YOLO v7 detection algorithm to obtain a detection target frame of the dynamic objects.

[0050] In step S302, the step of assigning values to the pixel points in the panoramic image based on the key scene information specifically includes: assigning high pixel values to the pixel points located at the feature points and the pixel points located in the neighborhood of the feature points in the panoramic image by using pixel assignment, and assigning low pixel values to all the pixel points located in the detection target frame of the dynamic objects; wherein the pixel value assigned to the pixel points located in the neighborhood of the feature points decreases with the increase of the neighborhood radius; if the same pixel point is assigned multiple times, the pixel point takes the maximum value assigned thereto.

[0051] In step S303, the step of projecting the field of view regions of the left and right two biomimetic eye cameras into the panoramic value image specifically includes: calculating the projection of the field of view regions of the left and right two biomimetic eye cameras on the panoramic value image of the panoramic camera by using the normalized plane projection equation of the biomimetic eye camera and the normalized spherical projection equation of the panoramic camera.

[0052] The specific step of merging the left field of view projection region and the right field of view projection region in step S303 is: taking the union of the left field of view projection region and the right field of view projection region to obtain a union region; and then generating a maximum circumscribed rectangle of the union region to obtain the current binocular field of view projection region.

[0053] The specific step of obtaining the value mean of the current binocular field of view projection region in the panoramic value image in step S304 is: according to the value assignment result of the pixel points in the panoramic value image, calculating the mean value of the value assignment of the pixel points in the current binocular field of view projection region to obtain the value mean of the current binocular field of view projection region.

[0054] The specific step of searching for the target binocular field of view projection region higher than the value threshold in the panoramic value image in step S305 is:

[0055] S401, generating a rectangular frame with the same size and the same position as the current binocular field of view projection region in the panoramic value image;

[0056] S402, translating the current rectangular frame from near to far by a step s, and calculating the value mean in the rectangular frame after each translation;

[0057] S403, comparing the value mean in the rectangular frame after translation with the value threshold: if the value mean in the rectangular frame after translation is higher than the value threshold, the rectangular frame after translation is recorded as the target binocular field of view projection region; if the value mean in the rectangular frame after translation is lower than or equal to the value threshold, returning to step S402 and updating the current rectangular frame to the rectangular frame after the last translation.

[0058] In step S306, the displacement is an angular displacement; the specific step of calculating the displacement required by the left and right biomimetic eye cameras is: using the normalized plane projection equation of the left or right biomimetic eye camera to calculate the current pose of the left or right biomimetic eye camera in the initial pose coordinate system thereof; using the normalized spherical surface to panoramic camera pixel plane projection equation of the panoramic camera to calculate the target pose of the field of view of the left or right biomimetic eye camera moving from the current binocular field of view projection region to the target binocular field of view projection region in the panoramic value image; and finally calculating the rotation matrix R CT between the two poses according to the current pose and the target pose of the left or right biomimetic eye camera, so as to obtain the displacement required by the field of view of the left or right biomimetic eye camera moving from the current binocular field of view projection region to the target binocular field of view projection region in the panoramic value image.

[0059] An electronic device includes a memory and a processor, the memory has a computer program stored thereon, and the processor implements the aforementioned channel self-attention mechanism based self-supervised image depth estimation method when executing the computer program.

[0060] A computer readable storage medium, the computer readable storage medium has a computer program stored thereon, the computer program is implemented when the processor executes the preceding channel self-attention mechanism based self-supervised image depth estimation method.

[0061] Compared with the existing method, the beneficial effects of the present application are:

[0062] 1) Constructing a self-supervised depth estimation network based on channel attention mechanism, using the sparse landmark points obtained by the SLAM system for network pre-training can reduce the optimization difficulty of the network and alleviate the slow convergence problem;

[0063] 2) In the training process of the self-supervised depth estimation network, the camera pose estimation task is handed over to the SLAM system, instead of using the pose estimation network to estimate the camera pose, and the camera pose obtained by the SLAM is used for reconstructing the target image, which can avoid the joint optimization of the depth estimation network and the pose estimation network, that is, only one depth estimation network needs to be optimized, which can reduce the burden of network training and reduce the difficulty of network learning;

[0064] 3) The self-supervised depth estimation network based on channel self-attention mechanism in the present application contains a structure perception module and a detail emphasis module, the structure perception module can help obtain better scene understanding and rich feature representation, and the detail emphasis module can highlight key local detail information, thereby forming a more accurate depth map and improving the subsequent mapping effect;

[0065] 4) The panoramic camera is used to shoot panoramic images to simulate the peripheral vision of the human eye, high-value images are obtained by custom assignment of the panoramic images, and then the binocular camera is guided by the gaze, which improves the ability of the system to obtain high-value information in the scene. Especially when performing the SLAM task, the problem of tracking loss faced by the SLAM system due to the texture sparsity and the presence of many dynamic objects in the images collected by the camera is avoided, thereby improving the stability of the SLAM system and enabling the visual SLAM method to realize accurate positioning and mapping in an unknown environment. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 It is a depth estimation network based on the channel self-attention mechanism architecture in embodiment 1;

[0067] Figure 2 It is a schematic diagram of the structure perception module in embodiment 1;

[0068] Figure 3 It is a schematic diagram of the detail emphasis module in embodiment 1;

[0069] Figure 4Flow chart for gaze guidance of binocular bionic eye camera using panoramic camera for example 4

[0070] Figure 5 Panoramic value map obtained for example 4, where the dark rectangle represents the detected dynamic object, and all pixels are assigned a low value;

[0071] Figure 6 Panoramic value map guiding the gaze control diagram, where the green rectangular frame represents the projection of the current gaze area of the left and right bionic eye; the red rectangular frame represents the target gaze area in the scene with a value higher than the value threshold and closest to the current gaze area;

[0072] Figure 7 SLAM algorithm framework for example 5

[0073] Figure 8 Minimization of left and right camera reprojection error in example 5

[0074] Figure 9 Local BA optimization in example 5

[0075] Figure 10 Octree diagram DETAILED DESCRIPTION

[0076] The present application will be described in detail below with reference to the accompanying drawings and specific examples. The present application is based on the technical solution of the present application and provides detailed implementation methods and specific operation processes, but the scope of protection of the present application is not limited to the following examples.

[0077] Example 1

[0078] Self-supervised image depth estimation method based on channel self-attention mechanism, as shown in Figure 1 , comprising:

[0079] Collecting images using binocular bionic eye cameras;

[0080] Defining one of the bionic eye cameras as the main camera, defining the target image as the main camera image, and defining the source image as the two adjacent frames of the main camera image and the image of the other bionic eye camera;

[0081] Inputting the images collected by the left and right bionic eye cameras into the SLAM system, and using the SLAM system to predict the camera pose and the sparse depth map of the target image;

[0082] inputting a target image and a sparse depth map obtained by a SLAM thread into a depth estimation network based on a channel self-attention mechanism architecture to obtain a scene depth map of the target image, wherein the depth estimation network based on the channel self-attention mechanism architecture comprises an encoder, a structure perception module and a decoder, and parameters of the depth estimation network based on the channel self-attention mechanism architecture are updated in the following manner:

[0083] inputting the target image into the encoder, the encoder adopting a ResNet-18 network as a backbone to extract semantic features, then inputting the semantic features into the structure perception module to generate new features and explicitly enhance the perception of scene structure; inputting the new feature map generated by the structure perception module into the decoder, the decoder first performing 3x3 convolution and up-sampling on the new feature map generated by the structure perception module, then entering a detail perception module, and performing 2x1 convolution and sigmoid function calculation on the feature map obtained by the detail perception module; obtaining a dense depth map of an original resolution after decoding;

[0084] inputting a sparse depth map obtained by a SLAM system into a last network structure of the decoder to participate in pre-training of the depth estimation network;

[0085] based on a relative pose between a target image frame and an adjacent frame or another camera in a binocular camera, projecting and reconstructing the target image by using a dense depth map of the target image predicted by the depth estimation network based on the channel self-attention mechanism and the relative pose between the frames, then constructing a reprojection error between the target image and the reconstructed target image, and minimizing the reprojection error during training.

[0086] As shown in Figure 2 , the operation process of the structure perception module comprises:

[0087] S101, a feature map generated by a ResNet-18 encoder is given Firstly, F is reshaped into wherein N=HxW is the number of pixels, then F is multiplied by a transpose matrix of F to calculate a feature similarity S

[0088] wherein i and j represent any two channels, S ij represents the feature similarity of the two channels;

[0089] S102, the similarity S is converted into a discriminability D by element-wise subtraction

[0090] D ij = max i (S)-S i,j , Dij represents the influence of the j-th channel on the i-th channel;

[0091] S103, a softmax layer is applied to obtain the attention map

[0092] A ij represents focusing attention on specific parts in two channels, extracting key information in the channels while ignoring irrelevant information;

[0093] S104, the transpose matrix of the attention map A and F is multiplied, and the result is reshaped into Let F and Element-wise sum operation is performed between the results to obtain the final output

[0094]

[0095] As shown in Figure 3 , the detail perception module restores the resolution by fusing the high-level feature H and the low-level feature L connected by skip connection. The specific operation process includes:

[0096] S201, first, the low-level feature L and the high-level feature H are connected, then a convolution layer is used and batch normalization processing is performed to obtain U, so as to balance the scale of the features:

[0097] Wherein, f() represents connection, represents 3x3 or 1x1 convolution, BN represents batch normalization processing, and ReLU is used as the activation function σ();

[0098] S202, U is compressed into a vector by global average pooling to obtain global context, and 2 1x1 convolution layers and sigmoid function are used to calculate the weight vector to recalibrate the channel features while measuring their importance:

[0099] Wherein, H and W are the height and width of U, and δ() represents the sigmoid function;

[0100] S203, element-wise multiplication is performed between V and U to generate the recalibrated feature, and the sum operation is performed between the recalibrated feature and U to obtain the final output O:

[0101] O = V⊙U + U, where ⊙ is element-wise dot product.

[0102] Embodiment 2

[0103] An electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the self-supervised image depth estimation method based on channel self-attention mechanism of embodiment 1 when executing the computer program.

[0104] Embodiment 3

[0105] A computer readable storage medium stores a computer program, and the computer program implements the self-supervised image depth estimation method based on channel self-attention mechanism of embodiment 1 when executed by a processor.

[0106] Embodiment 4 uses a panoramic camera to guide the gaze of a binocular biomimetic eye camera

[0107] The gaze of the binocular biomimetic eye camera is guided by the panoramic camera, and the flow is as shown in Figure 4 The binocular biomimetic eye camera collects high-value images under the guidance of the panoramic camera as input for the SLAM system in embodiment 1:

[0108] S301, a panoramic camera capable of collecting scene information around 360° is used to obtain a panoramic image; then key scene information affecting the running accuracy and robustness of SLAM, such as feature points and dynamic objects, is detected in the panoramic image.

[0109] Among them, the feature point extraction method is to extract using the FAST key point detection algorithm; when extracting feature points, the BRIEF descriptor calculation of the ORB feature points extracted by the SLAM algorithm is omitted because feature matching is not required. Further, the FAST is a corner point, mainly detects places where local pixel gray changes obviously, and is known for its speed. Its idea is: if a pixel is very different from its neighbors (too bright or too dark), it is more likely to be a corner. The detection process of the FAST key point detection algorithm is as follows:

[0110] (1) Select a pixel p in the image, and assume its brightness is I p ;

[0111] (2) Set a threshold T(I p 's 20%);

[0112] (3) Select 16 pixel points on a circle with a radius of 3 centered on pixel p;

[0113] (4) If there are N consecutive points on the selected circle whose brightness is greater than I p +T or less than I p -T, then the pixel p can be considered as a feature point (N is usually 12, i.e. FAST-12);

[0114] (5) Loop the above 4 steps and perform the same operation on each pixel.

[0115] The detection method of the dynamic object is to detect by using a YOLO v7 detection algorithm; the dynamic object is represented by a detection target frame. For dynamic object detection, the YOLO v7 detection algorithm is used to detect pedestrians, animals and other dynamic objects or potential dynamic objects that have a negative effect on map reuse in the panoramic image.

[0116] S302, value the pixel points in the panoramic image based on the key scene information to obtain a panoramic value image, as shown in Figure 5

[0117] Further, since the feature points are the objects that the SLAM algorithm focuses on, the feature points and their neighborhoods are assigned high gray values; the standard of the high gray value is a gray value higher than 240. For dynamic objects that seriously damage the performance of the SLAM algorithm, all pixel points in the detection target frame of the dynamic object are assigned low gray values, even lower than the area without detected feature points; the standard of the low gray value is a gray value lower than 80.

[0118] Further, since the feature points are the objects that the SLAM algorithm focuses on, the feature points and their neighborhoods are assigned high gray values; the standard of the high gray value is a gray value higher than 240. For dynamic objects that seriously damage the performance of the SLAM algorithm, all pixel points in the detection target frame of the dynamic object are assigned low gray values, even lower than the area without detected feature points; the standard of the low gray value is a gray value lower than 80.

[0119] S303, project the field of view regions of the left and right two biomimetic eye cameras into the panoramic value image respectively to obtain a left field of view projection region and a right field of view projection region; simultaneously control the left field of view projection region and the right field of view projection region to overlap; merge the left field of view projection region and the right field of view projection region to obtain a current binocular field of view projection region, as shown in the middle green rectangular frame. Figure 6

[0120] Further, the step of calculating the projection of the pixel planes of the left and right two biomimetic eye cameras on the pixel plane of the panoramic camera is specifically:

[0121] Further, the step of calculating the projection of the pixel planes of the left and right two biomimetic eye cameras on the pixel plane of the panoramic camera is specifically:

[0122] S501, calculate 4 pixel points u at the vertices of the pixel plane of the left biomimetic eye camera by using the normalized plane projection equation of the biomimetic eye camera​​L(0,0) L(m,0) L(m,n) L(0,n) The projection coordinates on the left biomorphic eye camera normalized plane are calculated according to the following formula,

[0123]

[0124]

[0125] In the above formula, p CN (·) is the normalized plane projection equation of the biomorphic eye camera; m and n are the maximum pixel coordinates of the biomorphic eye camera pixel plane W and H directions respectively;

[0126] S502, the aperture center of the panoramic camera is taken as the coordinate origin to establish a panoramic camera coordinate system, and the four projection coordinates obtained in step S501 are transformed into the panoramic camera coordinate system by using the following formula,

[0127]

[0128]

[0129] In the above formula, T L0P is the pose of the panoramic camera relative to the initial pose coordinate system of the left biomorphic eye camera (the initial pose coordinate system of the left biomorphic eye camera is established with the aperture center of the left biomorphic eye camera as the coordinate origin), and the transformation matrix is obtained by calibration; T L0L is the pose of the current pose of the left biomorphic eye camera relative to its initial pose coordinate system, and the transformation matrix is obtained by the biomorphic eye camera control feedback module;

[0130] S503, in the panoramic camera coordinate system, the coordinate origin of the initial pose coordinate system of the left biomorphic eye camera (the coordinate origin of the left biomorphic eye camera coordinate system is the aperture center of the left biomorphic eye camera) is connected with P P(0,0) , P P(m,0) , P P(m,n) and P P(0,n) respectively to obtain four projection straight lines L P1 , L P2 , L P3 and L P4 representing the four vertices of the field of view of the left biomorphic eye camera; the equation expressions of the four projection straight lines are,

[0131]

[0132] ​​​In the above formula, P1 represents P(0,0); P2 represents P(m,0); P3 represents P(m,n); P4 represents P(0,n); PL0 represents the origin of the coordinate system of the initial pose coordinate system of the left bionic eye camera; a, b, and c are constant parameters, all representing the direction number of the straight line direction vector;

[0133] The normalized spherical equation for the panoramic camera is shown below.

[0134] x 2 +y 2 +z 2 =1

[0135] By simultaneously solving the equations of the projected line and the normalized sphere, the intersection points of the projected line and the normalized sphere of the panoramic camera are obtained. Since each line intersects the sphere at two points, but because the bionic eye camera is limited by its range of motion and cannot observe the scene within the panoramic camera coordinate system x < 0, only the four intersection points P where x > 0 are retained. P1 P P2 P P3 and P P4 The intersection point P P1 P P2 P P3 and P P4 That is, the projection of the vertex of the pixel plane of the left bionic eye camera onto the normalized sphere of the panoramic camera;

[0136] S504, using the projection equation from the normalized sphere of the panoramic camera to its pixel plane, finds the intersection point P. P1 P P2 P P3 and P P4 Projecting the image onto the pixel plane of the panoramic camera, we obtain the intersection point P. P1 P P2 P P3 and P P4 The projection point; the projection equation ρ of the normalized sphere of the panoramic camera onto its pixel plane. B→I (·) is shown below,

[0137] u PIi =ρ B→I (P Pi ), (i = 1, 2, 3, 4)

[0138] S505, connecting the intersection point P P1 P P2 P P3 and P P4 The projection point is used to obtain the left field of view projection area (that is, the projection of the left bionic eye camera pixel plane onto the pixel plane of the panoramic camera, which is also the projection onto the panoramic value image).

[0139] S506, the right bionic eye camera can obtain the right field of view projection region according to the above steps S501 to S505.

[0140] It should be noted that the left field of view projection region and the right field of view projection region are irregular quadrilaterals, which is a distortion problem occurring in the spherical projection process of the panoramic camera. It is necessary to control the overlap of the left field of view projection region and the right field of view projection region. The steps are: using the bionic eye motion control module to control the motion of the left and right bionic eye cameras, and trying to overlap the left field of view projection region and the right field of view projection region, that is, controlling the left and right bionic eye cameras to approximately gaze at the same scene area. Further, although the left field of view projection region and the right field of view projection region cannot completely overlap, the left field of view projection region and the right field of view projection region can be controlled to overlap as much as possible.

[0141] Further, the left field of view projection region and the right field of view projection region are merged, and the steps are: taking the union of the left field of view projection region and the right field of view projection region to obtain a union region; then generating the maximum circumscribed rectangle of the union region to obtain the current binocular field of view projection region.

[0142] S304, obtaining the value mean of the current binocular field of view projection region in the panoramic value image, specifically: according to the assignment result of the pixel points in the panoramic value image, calculating the assignment mean of the pixel points in the current binocular field of view projection region to obtain the value mean of the current binocular field of view projection region.

[0143] Among them, the value threshold V th The number of successfully tracked feature points in the running process of the SLAM algorithm is selected, the purpose is to enable the motion control module to accurately make motion decisions. If the value mean V mc > V th , it means that the value of the current binocular field of view projection region scene can meet the demand of the good operation of the SLAM algorithm, that is, even if the binocular field of view projection region is moved to a region with higher value mean V mc , the promotion is also limited, and the shaking caused by the motion may affect the performance of the SLAM, at this time, the line of sight transfer motion is not selected; if the value mean V mc ≤ V th , it means that the value in the binocular field of view projection region scene at this time is insufficient to support the stable operation of the SLAM algorithm, which may be that the number of extractable feature points in the scene is insufficient or that the scene contains dynamic objects, at this time, the line of sight is transferred to the target binocular field of view projection region whose value mean is higher than the value threshold V th and closest to the current binocular field of view projection region.

[0144] S305, judging the size between the value mean of the current binocular field of view projection region and the value threshold: if the value mean of the current binocular field of view projection region is greater than the value threshold, the current binocular field of view projection region is not moved, and the high value images currently collected by the left and right bionic eye cameras are taken as the input of the SLAM system; if the value mean of the current binocular field of view projection region is less than or equal to the value threshold, a target binocular field of view projection region (such as the one shown in the middle red rectangular frame) with a value mean higher than the value threshold in the size of the binocular field of view projection region is searched in the panoramic value image. The binocular field of view projection region with a higher value can be searched by the following steps: Figure 6

[0145] S401, generating a rectangular frame with the same size and position as the current binocular field of view projection region in the panoramic value image;

[0146] S402, translating the current rectangular frame from near to far by a step s, and calculating the value mean in the rectangular frame after each translation;

[0147] S403, comparing the size between the value mean in the rectangular frame after translation and the value threshold: if the value mean in the rectangular frame after translation is higher than the value threshold, the rectangular frame after translation is recorded as the target binocular field of view projection region; if the value mean in the rectangular frame after translation is lower than or equal to the value threshold, returning to step S402, and updating the current rectangular frame to the rectangular frame after the last translation.

[0148] After obtaining the target binocular field of view projection region with a value mean higher than the value threshold, step S306 is entered:

[0149] S306, calculating the displacement amount required by the left and right bionic eye cameras according to the current binocular field of view projection region and the target binocular field of view projection region respectively; moving the left and right bionic eye cameras according to the displacement amount, and then taking the high value images currently collected by the left and right bionic eye cameras as the input of the SLAM system.

[0150] The displacement amount is an angular displacement amount, that is, the movement mode of the left and right bionic eye cameras is pure rotation without translation.

[0151] The step of calculating the angular displacement amount required by the left and right bionic eye cameras is: the current attitude of the left or right bionic eye camera in the initial pose coordinate system of the camera is calculated by using the normalized plane projection equation of the left or right bionic eye camera; the target attitude of the field of view of the left or right bionic eye camera is calculated by using the projection equation of the normalized spherical surface to the pixel plane of the panoramic camera, so that the field of view of the left or right bionic eye camera is moved from the current binocular field of view projection region to the target binocular field of view projection region in the panoramic value image; finally, the rotation matrix R CT ​The field of view of the left or right bionic eye camera can be obtained by moving the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area.

[0152] The specific steps for calculating the displacement amount required by the left and right bionic eye cameras are as follows:

[0153] S601, using the normalized spherical projection equation of the panoramic camera to the pixel plane of the panoramic camera B→I (·) to project the center pixel coordinates u PT of the target binocular field of view projection area in the panoramic value image onto the normalized sphere of the panoramic camera to obtain a spherical target point; then convert the spherical target point to the initial pose coordinate system of the left bionic eye camera to obtain the target pose P L0T of the left bionic eye camera, and the specific calculation formula is as follows,

[0154]

[0155] In the above formula, P L0T represents the target pose of the left bionic eye camera required to move the field of view of the left bionic eye camera from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area; u PT is the center pixel coordinate of the target binocular field of view projection area in the panoramic value image.

[0156] S602, using the normalized plane projection equation of the left bionic eye camera CN (·), the center point u LC = [m / 2, n / 2] T of the pixel plane of the left bionic eye camera is projected onto the normalized plane of the left bionic eye camera to obtain the current pose P L0C of the left bionic eye camera, and the specific calculation formula is as follows,

[0157]

[0158] In the above formula, P L0C represents the current pose of the left bionic eye camera in the initial pose coordinate system; u LC is the center point of the pixel plane of the left bionic eye camera.

[0159] S603, using the obtained current pose and target pose of the left bionic eye camera, calculate the rotation matrix R CT between the two poses, which is specifically:

[0160] First, according to the definition of the cross product, the rotation between two vectors is calculated as,

[0161]

[0162] Since the rotation axis in the rotation vector is a unit vector, the vector after the cross product in the above formula is normalized;

[0163] Then the rotation angle is calculated according to the following formula,

[0164]

[0165] Finally, according to the rotation axis n CT and the rotation angle θ CT , the rotation matrix R CT is obtained by using the Rodrigues formula,

[0166]

[0167] The angular displacement amount required for moving the field of view of the left bionic eye camera from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area can be calculated;

[0168] S604, the right bionic eye camera can obtain the angular displacement amount required for moving the field of view of the right bionic eye camera from the current binocular field of view projection area in the panoramic value image to the target binocular field of view projection area according to the above steps S601 to S603.

[0169] Further, the left and right bionic eye cameras are moved according to the displacement amounts: the yaw angle and the pitch angle of the left and right bionic eye cameras are controlled by motors respectively to move the left and right bionic eye cameras. The motors can be controlled by the rotation instructions issued by the bionic eye control module.

[0170] It should be noted that when the panoramic camera continues to shoot the next image, the left and right bionic eye cameras will repeat the above positioning method to reposition to obtain high-value images that can be input to the SLAM system.

[0171] Embodiment 5 SLAM thread

[0172] Referring to Figure 7 , the present application proposes a SLAM algorithm for bionic eye motion characteristics based on the improvement of ORB-SLAM2, which is used in embodiment 1 or 4, including four modules, which are tracking thread, local mapping thread, loop closure thread and global BA thread, specifically:

[0173] (1) Tracking thread

[0174] The main task of the tracking thread is to extract feature points from the input preprocessed image, estimate the camera pose based on feature matching, calculate the landmark depth, and determine whether the current frame is a key frame.

[0175] First, the ORB feature points are extracted in the input preprocessed image. After the images collected by the left and right biomimetic eye cameras are input into the SLAM algorithm framework, the ORB feature points of the two images are extracted in the preprocessing module, and then the input original image data is discarded. All subsequent algorithm operations are based on the extracted ORB feature points, the expression of the image is changed from a set of pixel points to a set of ORB feature points, and the data amount of the running cache is reduced. The extraction of the ORB feature points can be realized through OpenCV. First, the positions of the oriented FAST key points are detected, and then the BRIEF descriptor is calculated according to the key point positions.

[0176] Secondly, the left biomimetic eye is defined as the main camera, and the initial estimation of the pose of the camera is performed. The initial camera pose is estimated by using the constant speed model of the camera. It is assumed that the camera motion is uniform motion, the relative motion between each frame is constant, and the pose of the t frame is initialized based on the relative pose between the t-2 frame and the t-1 frame. Since the initial pose is assumed to be uniform motion, the actual camera will not move at a constant speed, and this assumption is relatively rough, and further optimization of the pose estimation is required through the tracking of the local map. The ORB feature points of the current left camera frame are retrieved in the local map for feature point matching, and then a PnP problem is constructed to solve the left camera pose by minimizing the re-projection error, so as to optimize the left camera pose.

[0177] Further, since the relative motion between the left and right biomimetic eye cameras of the designed biomimetic eye SLAM system cannot obtain accurate binocular landmark depth through binocular disparity method, the invention utilizes binocular matching, based on the feature point matching between the left and right cameras and the relative pose of the left and right cameras, to calculate the depth by triangulation to construct the landmark point. During the initialization of the binocular SLAM, the left and right biomimetic eye cameras stay at their initial poses, at this time, the pose T L0R0 between the initial coordinate systems of the left and right biomimetic eye cameras obtained through calibration is directly used to calculate the disparity based on binocular matching, so as to obtain the binocular landmark depth and construct the initial local map.

[0178] The initialization process of the binocular SLAM is very short, and only one frame is required to complete the initialization under the condition that the number of feature points is sufficient. For the requirement of supplementing the landmark points in the tracking process, the relative pose T LR between the left and right biomimetic eye cameras needs to be estimated first.

[0179] First, the relative poses T L0L and T R0R of the left and right biomimetic eye cameras relative to their initial coordinate systems and the pose T L0R0 between the initial coordinate systems of the left and right biomimetic eye cameras are calculated to obtain the initial value of the relative pose of the left and right biomimetic eye cameras in motion:

[0180]

[0181] Next, referring to Figure 8 , based on the matching of ORB feature points in the left and right cameras, and the matching of ORB feature points in the left camera and the road points, a PnP problem is constructed, the re-projection error of the right camera feature points is minimized, and the relative pose of the left and right cameras is optimized to obtain a more accurate T LR .

[0182] Using the optimized relative pose T LR of the left and right biomimetic eye cameras and the matched ORB feature points in the image, triangulation is performed to obtain a large number of high-quality road points. These road points greatly improve the accuracy and robustness of the tracking thread as a supplement to the road points in the local map.

[0183] Finally, the key frame selection condition is set to determine whether the current frame is set as a key frame. If the current frame meets any of the following conditions, the current frame is set as a key frame:

[0184] 1. The current frame is the first frame after the biomimetic eye line of sight control reaches the target gaze area;

[0185] 2. More than 13 frames from the last global repositioning;

[0186] 3. The local mapping thread is in an idle state;

[0187] 4. More than 15 frames from the last set key frame;

[0188] 5. The translational distance between the current frame and the last set key frame exceeds the threshold t th ;

[0189] 6. The number of successfully tracked feature points in the current frame is more than 70;

[0190] 7. The number of successfully tracked feature points in the current frame is less than 85% of the reference key frame.

[0191] (2) Local mapping thread

[0192] The local mapping thread manages the local map and performs local BA optimization. The purpose is to update and maintain the road points and key frames in the local map, and to perform local BA optimization while optimizing the pose of the key frames and the coordinates of the road points.

[0193] First, update the key frames in the local map. After the tracking thread determines to set the current frame as a key frame, the new key frame is associated according to its co-visibility relationship with the previous key frames, and the bag-of-words expression of the new key frame is calculated based on DBoW.

[0194] Further, the landmark points in the local map are updated and maintained. If a landmark point is to be kept in the local map, it must satisfy two conditions in the first three keyframes after it is created:

[0195] 1. More than 25% of the keyframes in which the landmark point is visible must be successfully tracked according to the pose prediction;

[0196] 2. If more than one keyframe is added after the landmark point is created, the landmark point must be observed by no less than three keyframes.

[0197] When the landmark point satisfies the two conditions, it will be removed only if the number of keyframes in which it is observed is less than 3 after that.

[0198] New landmark points are created based on the co-view relationship between keyframes, ORB feature matching, and depth calculation by triangulation. After the depth is calculated by triangulation, the landmark point is checked for whether the depth is positive and the reprojection error.

[0199] In addition, the local BA optimization is essentially to construct a PnP problem, but in the optimization, the coordinates of the landmark points are also added as optimization parameters. As shown in FIG. 1, the BA is explained from the perspective of graph optimization, wherein C1, C2, and C3 circles represent camera pose nodes, representing camera pose parameters to be optimized; P1, P2,..., and P7 circles represent landmark point nodes, representing three-dimensional coordinate parameters of the landmark points to be optimized, and the connection between the nodes represents an error term defined in the nonlinear optimization, which is defined as: Figure 9

[0200]

[0201] wherein u2 represents the observed coordinates of the 3D point P, and e represents the error between the projection coordinates of the 3D point P and the observed coordinates, i.e., the reprojection error.

[0202] In the iterative optimization, the derivative of the reprojection error e with respect to the camera pose is:

[0203]

[0204] wherein δξ represents the left perturbation of the camera pose T, and X', Y', and Z' represent the coordinates of the spatial 3D point transformed into the camera coordinate system.

[0205] The derivative of the reprojection error e with respect to the spatial 3D point P is:

[0206]

[0207] ​where R is the rotation matrix in the camera pose.

[0208] After obtaining the derivative of the re-projection error e with respect to the camera pose T and the spatial coordinates of the feature points P, when using the Gauss-Newton method or the Levenberg-Marquardt method to optimize the objective function, the gradient direction guides the update of the optimization variables, the camera pose T and the spatial coordinates of the feature points P, and the iteration is performed until the error of the objective function converges, and the optimal camera pose of the key frame and the coordinates of the road marker in the local map are obtained.

[0209] Finally, the local key frame is removed. When 90% of the road marker points in a key frame can be observed in at least three other key frames, the key frame is discarded as a redundant key frame. If the number of key frames is not limited, the data size in the local BA will become larger and larger, slowing down the optimization speed and affecting the real-time performance of the local mapping.

[0210] (3) Loop closure thread

[0211] The loop closure thread detects large loops based on DBoW2 and corrects the accumulated error of the camera trajectory by performing pose graph optimization. First, the loop is detected based on DBoW2 to find the loop candidate key frame, and the relative pose between the current key frame and the loop candidate key frame is calculated. Further, the loop is closed based on the co-visibility relationship, the loop is corrected based on the relative pose of the current key frame and the loop key frame calculated in the previous step, and pose graph optimization is performed.

[0212] Pose graph optimization only optimizes the camera pose, and the nodes in the pose graph are only camera pose nodes, while the edges connecting the nodes are the estimates of the relative poses between two camera pose nodes. Among them, the initial value of the pose node is the camera pose of each key frame before loop correction, and the edge is the relative pose between the camera poses of each key frame after loop correction. Suppose there are K i , K j two key frames, the camera poses before loop correction are T Wi and T Wj , and the relative pose between K i and K j after the camera pose of the key frame in the loop is corrected in the loop fusion is T ij , then the error e ij is:

[0213]

[0214] Based on the error, a nonlinear optimization is constructed, and the derivatives of the error terms with respect to T Wi and T Wj are solved for linearization, which can realize the optimization of the pose graph.

[0215] (4) Global BA thread

[0216] The global BA thread is started to obtain globally consistent camera poses and map structure. After the loop closure pose graph optimization is completed, the global BA optimization is performed in a separate thread to obtain the globally optimal solution. If a new loop is detected during the global BA optimization process, the global BA thread is terminated, and the global BA thread is started again after the loop closure pose graph optimization is completed. After the global BA thread optimization is completed, the updated keyframe camera poses are merged with the non-updated keyframe camera poses by generating a tree, and the coordinates of the landmarks are corrected according to the update of the reference keyframe camera poses.

[0217] Embodiment 6 Construction of error function

[0218] Embodiment 1 or 4 or 5, in training the depth estimation network based on the channel self-attention mechanism, or in other words, in parameter optimization, it is necessary to construct a reprojection error between the target image and the reconstructed target image, and the reprojection error is minimized during training.

[0219] This embodiment provides a specific construction scheme of the reprojection error function:

[0220] Target image I t is a left biomimetic eye camera image, the source image is two frames of left biomimetic eye camera images I s ∈{I t-1 ,I t+1} and right biomimetic eye camera image I s =I tr , and the pose of the target image relative to the source image is T st , which is obtained by the SLAM system.

[0221] Construct a photometric reprojection loss function L based on the reprojection error p , and minimize L p to update the weight parameters of the depth estimation network based on the channel self-attention mechanism, train the depth estimation network process, and define the photometric reprojection loss function L p :

[0222] Where pe() represents the photometric error composed of the L1 distance and the structural similarity SSIM, I s→t represents the reconstructed image.

[0223]

[0224] I s→t =I s <proj(D tT st ,K)>,

[0225] where proj() denotes the predicted depth map D t In I t , α = 0.85, <·> denotes bilinear interpolation sampling operation, K denotes camera intrinsic, D t denotes the predicted depth map, T st denotes the camera pose obtained by the SLAM system;

[0226] where μ x , μ y denotes the luminance mean, represents the brightness index in SSIM, σ x , σ y denotes the luminance standard deviation, represents the contrast index, σ xy is the luminance covariance, and the ratio of the product σ x σ y to the standard deviation represents the structure index, C1 and C2 are constants to avoid errors caused by the denominator being zero.

[0227] Meanwhile, the embodiment further defines an edge-aware smoothing loss L s when training the depth estimation network based on the channel self-attention mechanism:

[0228] where, is the mean normalized inverse depth;

[0229] The luminance reprojection loss L p and the edge-aware smoothing loss L s are combined as the final training loss function of the depth estimation network based on the channel self-attention mechanism:

[0230] where S is the scale number, and λ is the weight of the edge-aware smoothing loss term.

[0231] Embodiment 7 utilizes the prediction result to perform dense mapping

[0232] The embodiment further takes the depth map predicted by the depth estimation network based on the channel self-attention mechanism in embodiments 1, 4, 5 or 6 as input, writes an observed pixel with a depth d in the depth map into the occupancy probability of the corresponding spatial point, and constructs an octree map.

[0233] The octree map is to continuously and uniformly divide the three-dimensional space into eight blocks, and finally model the entire space as a collection of the smallest space partition units (voxels). Referring to Figure 10As shown, a cube is continuously divided into eight small cubes of the same size until a small enough cube is obtained. At this time, the original large cube is the root node in the octree data structure, and the smallest cube is the leaf node, and each cube divided into eight small cubes can be used as the child node of the divided cube.

[0234] In the octree map, each node stores an occupancy probability in the range of [0, 1] to express its own space occupancy information, 0 represents unoccupied, and 1 represents occupied. When the occupancy information of all child nodes of a parent node is consistent, the parent node does not need to be expanded, and only the occupancy probability of the parent node needs to be stored. In the extreme case, if the entire space is completely occupied, only the occupancy probability 1 of the root node needs to be stored. Therefore, this map expression form is much more compact than the point cloud map, and can very effectively reduce the storage space.

[0235] For the occupancy probability, its role is not only to express the space occupancy information, but also to be used for dynamic modeling of the map, so that the map can more accurately express the real scene. Each new observation updates the occupancy probability stored in the node. If the node is continuously observed to be occupied, the occupancy probability is increased, otherwise it is decreased. Therefore, the octree map can cope with dynamic changes in the scene by continuously updating the occupancy probability of the node.

[0236] In practical applications, the probability value cannot be continuously increased or decreased without limit, and must be limited to the interval [0, 1].

[0237] Therefore, the actual storage in the node is the logit function value q (probability logarithm value) of the probability p, that is:

[0238]

[0239] If the occupancy probability needs to be queried, its inverse transformation is used:

[0240]

[0241] With the logarithmic probability, the entire octree map can be updated according to the depth data. Assuming that a pixel with depth d is observed in the predicted dense depth map, it means that an occupancy data is observed at the space point corresponding to the depth value, and the line segment from the camera optical center to this point should be free of objects (otherwise it will be blocked). Using this information, the octree map can be updated well and can handle moving structures.

[0242] Further, considering the computational burden of the mapping module and the update frequency of the map, the mapping module of the SLAM system only performs dense mapping on key frames.

[0243] The dense mapping needs to first obtain the depth map D of the key frame in the SLAM system t Projecting to the world coordinate system, output the point cloud expression of the scene observed by the current key frame, update the node occupancy probability in the above octree map using the point cloud input, and obtain the dense mapping of the current key frame. Specifically:

[0244] First, obtain the input image I of the left biomimetic eye camera of the key frame from the SALM algorithm t And the pose T of the left biomimetic eye camera in the world coordinate system wc Then input the image I t After four times down-sampling, input the designed depth estimation network based on channel attention mechanism to obtain the dense depth map D t .

[0245] Then according to the camera projection model, project the pixel u t In the depth map D t To three-dimensional space, and convert to the world coordinate system:

[0246]

[0247] After projecting all the depth estimated pixel points in the dense depth map D of the key frame t In the world coordinate system, the point cloud expression of the scene observed by the current key frame is obtained, and the point cloud is input into the octree map to update the node occupancy probability in the map, realizing the dense mapping of the current key frame.

[0248] With the continuous movement of the biomimetic eye camera, new key frames are constantly inserted, and the octree map of the scene can be continuously and dynamically updated, and finally the complete octree map of the explored scene is obtained. The obtained octree map stores the spatial occupancy information of the scene, and through continuous updating, the traces left by dynamic objects in the map are eliminated, so that the robot can obtain the necessary information required for navigation and obstacle avoidance functions in the modeled scene based on the map.

Claims

1. A self-supervised image depth estimation method based on channel self-attention mechanism, comprising the following steps: Images are acquired using a binocular bionic eye camera; Define one of the bionic eye cameras as the main camera, define the target image as the main camera image, and define the source images as the two main camera images adjacent to the target image and the other bionic eye camera image; Images acquired by the left and right bionic eye cameras are input into the SLAM system, and the SLAM system is used to predict the camera pose and the sparse depth map of the target image. The target image and the sparse depth map obtained by the SLAM thread are input into a depth estimation network based on a channel self-attention mechanism architecture to obtain a scene depth map of the target image. The depth estimation network based on the channel self-attention mechanism architecture includes an encoder, a structure-aware module, and a decoder. The parameters of the depth estimation network based on the channel self-attention mechanism architecture are updated in the following manner: The target image is input into the encoder, which uses a ResNet-18 network as its backbone to extract semantic features. These semantic features are then input into the structure-aware module to generate new features. The new feature map generated by the structure-aware module is input into the decoder. The decoder first performs a 3×3 convolution and upsampling on the new feature map generated by the structure-aware module, and then it enters the detail-aware module. The feature map obtained by the detail-aware module is then subjected to two 1×1 convolutions and a sigmoid function calculation. After decoding, a dense depth map at the original resolution is obtained. The sparse depth map obtained by the SLAM system is input into the last layer of the decoder network structure to participate in the pre-training of the depth estimation network. Based on the relative pose between the target image frame and adjacent frames or another camera in a stereo camera, the target image is projected and reconstructed using the dense depth map of the target image predicted by the depth estimation network based on the channel self-attention mechanism and the relative pose between frames. Then, a reprojection error is constructed between the target image and the reconstructed target image, and this reprojection error is minimized during training.

2. The image depth estimation method as described in claim 1, characterized in that, The computation process of the structure sensing module includes: S101, given the feature map generated by the ResNet-18 encoder First, reshape F as... Where N = H × W is the number of pixels, then F is multiplied by the transpose of F to calculate the feature similarity. Where i and j represent any two channels, S ij Indicates the feature similarity between two channels; S102, similarity S is converted into discriminant value through element-wise subtraction. D ij = max i (S)-S ij , D ij denotes the influence of the jth channel on the ith channel; S103, apply a softmax layer to obtain the attention map. A ij This means focusing attention on specific parts of two channels, extracting key information from the channels while ignoring irrelevant information; S104, the attention map A and the feature map F matrix are multiplied, and the result is reshaped into... Let F and The results are summed element-wise to obtain the final output.

3. The image depth estimation method as described in claim 1, characterized in that, The detail-aware module restores resolution by fusing high-level features H and low-level features L that skip connections. The specific computation process includes: S201 first concatenates the low-level feature L and the high-level feature H, then uses a convolutional layer and performs batch normalization to obtain U, in order to balance the scale of the features: Where f() represents a connection, This represents a 3×3 or 1×1 convolution, BN represents batch normalization, and ReLU is used as the activation function σ(). S202 compresses U into a vector using global average pooling to obtain the global context, and uses two 1×1 convolutional layers and a sigmoid function to compute the weight vector. To recalibrate channel characteristics while simultaneously measuring their importance: Where H and W are the height and width of U, and δ() represents the sigmoid function; S203, perform element-wise multiplication between V and U to generate a reweighted feature, then sum the reweighted feature with U to obtain the final output O: O = V⊙U + U, where ⊙ is the element-wise dot product.

4. The image depth estimation method as described in claim 1, characterized in that, Define the left camera in the binocular biomimetic eye camera as the main camera, the target image I t is the left biomimetic eye camera image, the source image is the two frames of left biomimetic eye camera images adjacent before and after the target image I s ∈{I t-1 ,I t+1} and the right biomimetic eye camera image I s = tr , the pose of the target image relative to the source image is T st , which is obtained by the SLAM system; A photometric reprojection loss function L is constructed based on a reprojection error p minimizing L p to update the weight parameters of the deep estimation network based on the channel self-attention mechanism, the photometric reprojection loss function L p is defined as: Where pe() represents the photometric error composed of L1 distance and structural similarity SSIM, I s→t The image represents the reconstructed image; I s→t =I s <proj(D t ,T st ,K)>, Where proj() represents the predicted depth map D t in I t In the 2D coordinates, α = 0.85, <·> represents the bilinear interpolation sampling operation, K represents the camera intrinsic parameters, and D t T represents the predicted depth map. st This represents the camera pose obtained by the SLAM system; Where, μ x μ y σ represents the mean luminance, which is the luminance index in SSIM. x , σ y σ represents the standard deviation of luminance and the contrast ratio. xy The photometric covariance is σ, which is the product of the photometric covariance and the standard deviation. x σ y The ratio represents the structural index, with C1 and C2 being constants to avoid errors caused by a denominator of 0.

5. The image depth estimation method as described in claim 4, characterized in that, During the training of the deep estimation network based on the channel self-attention mechanism, an edge-aware smoothing loss L is also defined. s : in, The mean-normalized inverse depth; Photometric reprojection loss L p With edge-aware smoothing loss L s Combined with the final training loss function of the deep estimation network based on the channel self-attention mechanism: Where S is the scale number and λ is the weight of the edge-aware smoothing loss term.

6. The image depth estimation method as described in any one of claims 1-5, characterized in that, A panoramic camera is used to guide the gaze of a binocular bionic eye camera, acquiring high-value images as input to the SLAM system. S301 acquires panoramic images through a panoramic camera and detects key scene information in the panoramic images that affects the execution of the SLAM task. S302, assign values ​​to pixels in the panoramic image based on key scene information to obtain a panoramic value image; S303 projects the field of view of the two bionic eye cameras onto the panoramic value image, respectively, to obtain the left field of view projection area and the right field of view projection area; at the same time, it controls the overlap of the left field of view projection area and the right field of view projection area. Merge the left and right field-of-view projection areas to obtain the current binocular field-of-view projection area; S304, obtain the average value of the current binocular field of view projection area in the panoramic value image; S305, determine the value of the current binocular field of view projection area and the value threshold: if the value of the current binocular field of view projection area is greater than the value threshold, do not move the current binocular field of view projection area, and use the high-value images currently acquired by the left and right bionic eye cameras as the input of the SLAM system; if the value of the current binocular field of view projection area is less than or equal to the value threshold, search for a target binocular field of view projection area with a higher value than the value threshold in the panoramic value image, and then proceed to step S306; S306, calculate the required displacement of the left and right bionic eye cameras based on the current binocular field of view projection area and the target binocular field of view projection area respectively; move the left and right bionic eye cameras according to the displacement amount respectively, and finally use the high-value images collected by the left and right bionic eye cameras in the target binocular field of view projection area as the input of the SLAM system.

7. The image depth estimation method as described in claim 6, characterized in that, The key scene information includes feature points and dynamic objects; the feature points are extracted using the FAST key point detection algorithm. The detection process for the dynamic object involves using the YOLO v7 detection algorithm to detect the dynamic object and obtain its detection bounding box.

8. The image depth estimation method as described in claim 6, characterized in that, In step S302, the step of assigning values ​​to pixels in the panoramic image based on key scene information specifically involves: assigning high pixel values ​​to pixels located at feature points and pixels located in the neighborhood of feature points in the panoramic image, and assigning low pixel values ​​to all pixels located within the detection target box of dynamic objects; wherein, the pixel value assigned to pixels located in the neighborhood of feature points decreases as the neighborhood radius increases; if the same pixel is assigned multiple values, the pixel takes the maximum value assigned to it.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the self-supervised image depth estimation method based on the channel self-attention mechanism as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the self-supervised image depth estimation method based on the channel self-attention mechanism as described in any one of claims 1-8.

Images