A neural network-based lightweight depth estimation method and system

Abstract

The present application relates to a kind of neural network-based lightweight depth estimation method and system, its method includes: step S1: data preprocessing;S2: depth estimation neural network is built;S3: pose estimation network is built;Two networks S4 are trained by setting loss function;According to the global voxel result generated according to the output of depth estimation and pose estimation result.This method provides method for the first time lightweight depth estimation network is applied to endoscopy image, and the network combines convolution and attention mechanism.The method uses the mixture of convolution and attention mechanism as encoder to aggregate local texture information and global contour feature.The method obtains competitive result, while also reducing the parameter quantity of depth estimation network.Compared with previous method, the proposed pose network obtains minimum error on dataset.

Description

Technical Field

[0001] This invention relates to the field of augmented / mixed reality, and more specifically to a lightweight depth estimation method and system based on neural networks. Background Technology

[0002] With the rapid development of AR / MR technology, its application in various industries is deepening. 3D reconstruction and depth estimation are crucial tasks in AR / MR, with considerable application potential in robot control, human-computer interaction, assisted medical care, and augmented reality. Early reconstruction methods based on multi-view geometry theory included Shape from Shading (SfS), Shape from Motion (SfM), and Simultaneous Localization and Mapping (SLAM). However, these geometry-based methods perform poorly in endoscopic environments. First, geometry-based methods heavily rely on feature extraction and feature matching. However, because human soft tissue is easily deformable and reflective, feature matching can only yield sparse matching points. Second, most geometry-based methods rely on illumination invariance, meaning the ambient illumination remains constant throughout the reconstruction process. However, there is no light source in the human body; the only light source is the illumination lamp installed in front of the endoscope lens. The light source changes with the movement of the lens, violating illumination invariance.

[0003] Deep learning-based depth estimation methods can be divided into two categories: (1) methods using visual cues, which utilize self-supervised learning strategies and use only endoscopic image sequences as input to estimate the 3D information corresponding to each input image. For example, a residual network is used as an encoder to construct a convolution-based decoder, and the depth corresponding to each pixel is estimated through an encoder-decoder U-shaped structure. (2) Methods using auxiliary data, which construct endoscopic simulation data through CT scans or models, use supervised learning methods to learn the geometry of the simulation data, and then use transfer learning methods to estimate the 3D depth information of the real endoscopic scene. The difference between simulation data and real data has always been the main reason hindering the development of performance in this type of method. The method using visual cues, which directly uses real endoscopic images as input and supervision, is currently more widely accepted by most researchers. For navigation applications, depth estimation networks not only need to ensure accuracy but also need to be integrated with other modules such as registration. Therefore, an effective and lightweight network structure is an important topic in self-supervised depth estimation. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a lightweight depth estimation method and system based on neural networks.

[0005] The technical solution of this invention is: a lightweight depth estimation method based on neural networks, comprising:

[0006] Step S1: Data preprocessing, including converting video data into image sequences, segmenting the image sequences, and performing data cleaning and data augmentation;

[0007] Step S2: Build a depth estimation network based on convolution and attention mechanisms. Take a single image as input and output the depth information corresponding to the image. Among them, layer processing based on convolution operation is used to obtain local texture features, and layer processing based on attention mechanism operation is used to obtain global contour features.

[0008] Step S3: Construct a pose estimation network based on convolution and multi-head attention modules. Take two cascaded adjacent images as input and output the transformation matrix between adjacent images. First, convolution is used to process the images to obtain image features, and then multi-head attention modules are used to process the image features for feature enhancement. At the same time, a loss function is set to constrain the two neural networks, the depth estimation network and the pose estimation network.

[0009] Step S4: Recover the point cloud based on the camera's intrinsic parameters and the depth information generated from the single image in Step S2; merge multiple point clouds into a global voxel based on the transformation matrix obtained from the pose estimation network in Step S3, and display the final result.

[0010] Furthermore, step S2: building a depth estimation network based on convolution and attention mechanisms, taking a single image as input, and outputting the depth information corresponding to the image, specifically includes:

[0011] Step S21: Perform pyramid downsampling on the input image I to obtain thumbnails I1, I2, and I3 respectively; I1, I2, and I3 are thumbnails of the original image after downsampling at different resolutions.

[0012] Step S22: The depth estimation network uses four encoders to receive the original images I, I1, I2 and I3 respectively, which are the dimensionality reduction encoder, the low-dimensional texture encoder, the contour encoder and the high-dimensional texture encoder; wherein the dimensionality reduction encoder receives the original image I, the low-dimensional texture encoder receives I1, the contour encoder receives I2 and the high-dimensional texture encoder receives I3.

[0013] The dimension reduction encoder receives the original image I and encodes the image to obtain a low-dimensional feature map F0.

[0014] The low-dimensional texture encoder encodes low-dimensional feature maps F0 and I1 to obtain feature map F1. The low-dimensional texture encoder comprises a dimensionality-reducing convolution, two types of convolution modules, and an attention mechanism module with an attention function. The two types of convolution modules are stacked alternately to reduce the number of parameters. The low-dimensional feature maps F0 and I1 are concatenated and input into the dimensionality-reducing convolution to generate a dimensionality-reduced feature S0. S0 then passes through multiple convolution modules and an attention mechanism module with an attention function to generate feature map F1.

[0015] The contour encoder encodes feature maps F1, S0, and I2 to obtain feature map F2. The contour encoder consists of two dimensionality-reducing convolutions, multiple attention modules with pooling, and two attention mechanism modules with attention functions. Features F1, S0, and I2 are concatenated and input into the dimensionality-reducing convolution to generate dimensionality-reduced feature S1. S1 is then passed through multiple attention mechanism modules with pooling, another dimensionality-reducing convolution, and two attention mechanism modules with attention functions to generate feature map F2.

[0016] The high-dimensional texture encoder encodes feature maps F2, S1, and I3 to obtain feature map F3. The high-dimensional texture encoder consists of dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function. Two types of convolutional modules are stacked alternately in the high-dimensional texture encoder to reduce the number of parameters. Feature maps F2, S1, and I3 are concatenated, undergoing dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function to generate feature map F3.

[0017] Step S23: The depth estimation network uses three decoders to receive the outputs of encoders of different dimensions, namely a high-dimensional decoder, a medium-dimensional decoder, and a low-dimensional decoder.

[0018] The high-dimensional decoder receives the output F3 and output U3 of the high-dimensional texture encoder, and predicts the low-resolution depth information D3.

[0019] The mid-dimensional decoder receives the output F2 of the contour encoder and the output U3 of the high-dimensional decoder, outputs U2, and predicts the mid-resolution depth information D2.

[0020] The low-dimensional decoder receives the output F1 of the low-dimensional texture encoder and the output U2 of the mid-dimensional decoder, outputs feature U1, and predicts high-resolution depth information D1.

[0021] Furthermore, a pose estimation network is constructed based on convolutional and multi-head attention modules. It takes cascaded adjacent images as input and outputs a transformation matrix between adjacent images. First, a convolutional module is used to process the images to obtain image features, and then a multi-head attention module is used to process the image features for feature enhancement. Simultaneously, a loss function is set to constrain the two neural networks, the pose estimation network and the depth estimation network, specifically including:

[0022] Step S31: Assume the original image is I k , the original image I k and adjacent image I k+1 Cascaded to form cascaded input I p ;

[0023] Step S32: The pose estimation network uses three feature extractors to process the cascaded input I. p Layer-by-layer feature extraction and enhancement are performed, namely, low-dimensional feature extraction enhancer, high-dimensional feature extraction enhancer, and high-dimensional feature extractor;

[0024] Among them, the high-dimensional feature extractor consists of convolution operations, while both the low-dimensional feature extraction enhancer and the high-dimensional feature extraction enhancer consist of convolution operations and multi-head attention operations.

[0025] The low-dimensional feature extraction enhancer receives input I. p Encode the cascaded images to output low-dimensional features P1;

[0026] The high-dimensional feature extraction enhancer receives low-dimensional feature P1, encodes P1, and extracts high-dimensional feature P2.

[0027] The high-dimensional feature extractor receives high-dimensional feature P2, encodes P2, and outputs pose feature P;

[0028] Step S33: Generate a vector by convolution based on the pose feature P, and then convert it into a transformation matrix;

[0029] Step S34: Based on the structure and objective of the pose estimation network and depth estimation network, define the loss function, including a similarity loss based on photometric correction. and edge smoothing loss

[0030] Among them, similarity loss based on photometric correction The similarity loss is calculated using a structural similarity index, as shown in formula (1):

[0031]

[0032] Wherein, SSIM is the structural similarity index calculation function. It is an image that has been photometrically corrected. It is an image back-projected based on predicted depth information, camera parameters, and predicted pose; α is set to 0.85;

[0033] The edge smoothing loss As shown in formula (2):

[0034]

[0035] Where d is the predicted depth, It is the gradient of d in the x-direction. It is the gradient of d in the y direction. It is the gradient of I in the x-direction. It is the gradient of I in the y-direction, where I is the input image.

[0036] Furthermore, step S4: reconstructing the point cloud based on the camera's intrinsic parameters and the depth information generated from a single image, fusing multiple point clouds into a global point cloud based on the transformation matrix obtained from pose estimation, and displaying the final result, specifically includes:

[0037] The method for recovering point clouds based on camera intrinsic parameters and depth information generated from a single image is shown in the formula:

[0038]

[0039] Among them, (uv 1) T These are homogenized image coordinates, (xyz). T It represents the recovered 3D coordinates, d is the depth estimated by the network, and f is the depth. x and f y It refers to the camera's horizontal and vertical focal lengths, c x and c y This represents the offset of the camera's optical axis in the image coordinate system;

[0040] Based on the series of transformation matrices M1, ..., M1 obtained from the pose estimation network n Multiple point clouds are fused using a truncated signed distance function (TSDF), where the depth map D is used as the fusion point. t and transformation matrix M t The TSDF value and weight of the current frame are respectively represented by g t and w t This indicates that the TSDF values ​​and weights of the global voxels before the update are represented by G. t-1 and W t-1 This indicates that the updated global voxel TSDF values ​​and weights are represented by G. t and W t This means that multiple depths are fused into a single global voxel, as shown in the formula:

[0041]

[0042] W t =W t-1 +w t .

[0043] This invention also provides a lightweight depth estimation system based on neural networks, comprising the following modules:

[0044] The data preprocessing module is used to convert video data into image sequences, segment the image sequences, perform data cleaning, and data enhancement.

[0045] The depth estimation module is built based on convolution and attention mechanisms. It takes a single image as input and outputs the depth information of the image. Specifically, the convolution module is used to process the layered image to obtain local texture features, and the attention mechanism module is used to process the layered image to obtain global contour features, thus estimating the depth of each pixel in the input image.

[0046] The pose estimation module is built based on convolution and multi-head attention modules. It takes two concatenated adjacent images as input, first uses the convolution module to process the images to obtain image features, and then uses the multi-head attention module to process the image features for feature enhancement, and outputs the transformation matrix between adjacent images.

[0047] The point cloud fusion module reconstructs point clouds based on camera intrinsic parameters and depth information generated from single images; it then merges multiple point clouds into a single global voxel based on the transformation matrix obtained from pose estimation, and displays the final result.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. This invention discloses a lightweight depth estimation method based on neural networks. It designs a lightweight depth estimation network that combines convolution and attention mechanisms, which achieves competitive results while reducing the number of parameters.

[0050] 2. The method disclosed in this invention is a pose estimation method based on convolution and multi-head attention modules, which reduces errors compared with other methods and has good robustness.

[0051] 3. The method disclosed in this invention uses only endoscopic images as input, which can not only reduce the parameters of the network model while ensuring performance, but also improve the inference speed of the depth estimation network. Attached Figure Description

[0052] Figure 1This is a flowchart of a lightweight depth estimation method based on neural networks in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the depth estimation network in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the convolution structure in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the attention mechanism structure in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the multi-head attention structure in an embodiment of the present invention;

[0057] Figure 6 The image shows a comparison of the performance of a lightweight depth estimation method based on neural networks, as an embodiment of the present invention.

[0058] Figure 7 This is a block diagram of a lightweight depth estimation system based on a neural network, as described in an embodiment of the present invention. Detailed Implementation

[0059] This invention provides a lightweight depth estimation method based on neural networks. It designs a lightweight depth estimation network that combines convolution and attention mechanisms, achieving competitive results while reducing the number of parameters. It utilizes a pose estimation method based on convolution and multi-head attention modules, which reduces errors compared to other methods and exhibits good robustness. By using only endoscopic images as input, it not only reduces the number of network model parameters while maintaining performance but also improves the inference speed of the depth estimation network.

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0061] The embodiments of this invention use Python and PyTorch as the programming languages, and the devices used are RTX3090ti, Intel(R) i78750K CPU (2.21GHz, 6 cores) and 16GB RAM, running on Ubuntu 20.04.

[0062] Example 1

[0063] like Figure 1 As shown in the figure, an embodiment of the present invention provides a lightweight depth estimation method based on neural networks, which includes the following steps:

[0064] A lightweight depth estimation method using neural networks, characterized by comprising:

[0065] Step S1: Data preprocessing, including converting video data into image sequences, segmenting the image sequences, and performing data cleaning and data augmentation;

[0066] Step S2: Build a depth estimation network based on convolution and attention mechanisms. Take a single image as input and output the depth information corresponding to the image. Among them, layer processing based on convolution operation is used to obtain local texture features, and layer processing based on attention mechanism operation is used to obtain global contour features.

[0067] Step S3: Construct a pose estimation network based on convolution and multi-head attention modules. Take two cascaded adjacent images as input and output the transformation matrix between adjacent images. First, convolution is used to process the images to obtain image features, and then multi-head attention modules are used to process the image features for feature enhancement. At the same time, a loss function is set to constrain the two neural networks, the depth estimation network and the pose estimation network.

[0068] Step S4: Recover the point cloud based on the camera's intrinsic parameters and the depth information generated from the single image in Step S2; merge multiple point clouds into a global voxel based on the transformation matrix obtained from the pose estimation network in Step S3, and display the final result.

[0069] Step S2: Construct a depth estimation network based on convolution and attention mechanisms, taking a single image as input and outputting the 3D depth information corresponding to the image, specifically including:

[0070] Step S21: Perform pyramid downsampling on the input image I to obtain thumbnails I1, I2, and I3 respectively; I1, I2, and I3 are thumbnails of the original image after downsampling at different resolutions.

[0071] Step S22: The neural network uses four encoders to receive the original images I, I1, I2 and I3 respectively, which are a dimensionality reduction encoder, a low-dimensional texture encoder, a contour encoder and a high-dimensional texture encoder; wherein the dimensionality reduction encoder receives the original image I, the low-dimensional texture encoder receives I1, the contour encoder receives I2 and the high-dimensional texture encoder receives I3.

[0072] Figure 2 This is a schematic diagram of the depth estimation network in an embodiment of the present invention;

[0073] The dimensionality reduction encoder includes 3×3 convolution operations, while the low-dimensional texture encoder and high-dimensional texture encoder consist of multiple convolutional modules and an attention mechanism module. The convolutional modules are as follows: Figure 3 As shown, it is mainly used to extract local texture information, and the attention mechanism module is as follows: Figure 4As shown, it is mainly used to extract global texture information. The contour encoder consists of multiple attention mechanism modules, such as... Figure 4 As shown, it is mainly used to extract contour information. Both the low-dimensional texture encoder and the contour encoder contain a downsampling module, which generates downsampling features S0 and S1, respectively;

[0074] The dimension reduction encoder receives the original image I and encodes the image to obtain the feature map F0.

[0075] The low-dimensional texture encoder encodes low-dimensional feature maps F0 and I1 to obtain feature map F1. The low-dimensional texture encoder comprises a dimensionality-reducing convolution, two types of convolutional modules, and an attention mechanism module with an attention function. The two types of convolutional modules are stacked alternately in the low-dimensional texture encoder to reduce the number of parameters. The low-dimensional features F0 and I1 are concatenated and input into the dimensionality-reducing convolution to generate a dimensionality-reduced feature S0. S0 then passes through multiple convolutional modules and an attention mechanism module with an attention function to generate feature map F1.

[0076] The contour encoder encodes low-dimensional feature maps F1, S0, and I2 to obtain feature map F2. The contour encoder consists of two dimensionality-reducing convolutions, multiple attention modules with pooling, and two attention mechanism modules with attention functions. Features F1, S0, and I2 are concatenated and input into the dimensionality-reducing convolution to generate dimensionality-reduced feature S1. S1 is then passed through multiple attention mechanism modules with pooling, another dimensionality-reducing convolution, and two attention mechanism modules with attention functions to generate feature map F2.

[0077] The high-dimensional texture encoder encodes feature maps F2, S1, and I3 to obtain feature map F3. The high-dimensional texture encoder consists of dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function. Two types of convolutional modules are stacked alternately in the high-dimensional texture encoder to reduce the number of parameters. Feature maps F2, S1, and I3 are concatenated, undergoing dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function to generate feature map F3.

[0078] Step S23: The depth estimation network uses three decoders to receive the outputs of encoders of different dimensions, namely a high-dimensional decoder, a medium-dimensional decoder, and a low-dimensional decoder.

[0079] The high-dimensional decoder receives the output F3 from the high-dimensional texture encoder and outputs U3.

[0080] The mid-dimensional decoder receives the outputs U3 from the contour encoder F2 and the high-dimensional decoder, and outputs U2;

[0081] The low-dimensional decoder receives the output F1 of the low-dimensional texture encoder and the output U2 of the mid-dimensional decoder, and outputs features.

[0082] Step S24: The three decoders of the neural network also decode the cascaded feature maps obtained by encoders of different dimensions;

[0083] The high-dimensional decoder, medium-dimensional decoder and low-dimensional decoder have the same structure, consisting of a convolutional layer, a linear upsampling layer and a sigmoid layer;

[0084] The high-dimensional decoder decodes the transmitted feature map and predicts the low-resolution depth information D3.

[0085] The mid-dimensional decoder decodes the high-dimensional feature map and predicts the mid-resolution depth information D2.

[0086] The low-dimensional decoder decodes the mid-dimensional feature map and predicts high-resolution depth information D1.

[0087] A pose estimation network is built based on convolutional and multi-head attention modules. It takes two cascaded adjacent images as input and outputs a pose matrix between adjacent images. First, a convolutional module is used to process the images to obtain image features, and then a multi-head attention module is used to process the feature maps for feature enhancement. Simultaneously, a loss function is set to constrain the two neural networks, specifically including:

[0088] Step S31: Transfer the original image I k and adjacent images (I k+1 Cascaded input I p ;

[0089] Step S32: The neural network uses three feature extractors to perform layer-by-layer feature extraction and enhancement on the cascaded input, namely a low-dimensional feature extraction enhancer, a high-dimensional feature extraction enhancer, and a high-dimensional feature extractor;

[0090] The high-dimensional feature extractor consists of convolutional operations, while the low-dimensional and high-dimensional feature extraction enhancers both consist of convolutional and multi-head attention operations. The feature map obtained after convolution is denoted by F, and the output of the multi-head attention is... The calculation process for multi-head attention is shown in formula (1):

[0091]

[0092] Where Q,K,v are matrices projected from the features.

[0093] The low-dimensional feature extraction enhancer receives input I. p Encode the cascaded images to output low-dimensional features P1;

[0094] The high-dimensional feature extraction enhancer receives low-dimensional feature P1, encodes P1, and extracts high-dimensional feature P2.

[0095] The high-dimensional feature extractor receives high-dimensional feature P2, encodes P2, and outputs pose feature P;

[0096] Figure 5 A schematic diagram of a multi-head attention structure in an embodiment of the present invention is shown.

[0097] Step S33: Generate a vector by convolution based on the pose feature P, and then convert it into a transformation matrix;

[0098] Step S34: Based on the structure and objective of the pose estimation network and depth estimation network, define the loss function, including a similarity loss based on photometric correction. and edge smoothing loss

[0099] Among them, similarity loss based on photometric correction The structural similarity index is used to calculate the similarity loss of two-dimensional images, transforming the three-dimensional reconstruction problem into a similarity problem. The similarity loss is shown in Equation (2).

[0100]

[0101] Wherein, SSIM is the structural similarity index calculation function. It is an image that has been photometrically corrected. It is an image back-projected based on predicted depth information, camera parameters, and predicted pose;

[0102] The edge smoothing loss As shown in formula (3):

[0103]

[0104] Where d is the predicted depth, It is the gradient of d in the x-direction. It is the gradient of d in the y direction. It is the gradient of I in the x-direction. It is the gradient of I in the y-direction, where I is the input image.

[0105] Step S4: Recover the point cloud based on the camera's intrinsic parameters and the depth information generated from a single image; merge multiple point clouds into a global point cloud based on the transformation matrix obtained from pose estimation; and display the final result. Specifically, this includes:

[0106] The method for recovering point clouds based on camera intrinsic parameters and depth information generated from a single image is shown in the formula:

[0107]

[0108] Among them, (uv 1) T These are homogenized image coordinates, (xyz). T It represents the recovered 3D coordinates, d is the depth estimated by the network, and f is the depth. x and f y It refers to the camera's horizontal and vertical focal lengths, c x and c y This represents the offset of the camera's optical axis in the image coordinate system.

[0109] A series of transformation matrices M1, ..., M2 can be obtained from the pose estimation network. n A truncated signed distance function (TSDF)-based algorithm is used to fuse multiple point clouds. This is achieved through the fusion of depth maps D. t and pose M t The TSDF value and weight of the current frame are respectively represented by g t and w t This indicates that the TSDF values ​​and weights of the global voxels before the update are represented by G. t-1 and W t-1 This indicates that the updated global voxel TSDF values ​​and weights are represented by G. t and W t This means that multiple depths are fused into a single global voxel, as shown in the formula:

[0110]

[0111] W t =W t-1 +w t

[0112] Figure 6 This is a comparison chart showing the performance of a lightweight depth estimation method based on neural networks, as an embodiment of the present invention.

[0113] This invention discloses a lightweight depth estimation method based on neural networks. It designs a lightweight depth estimation network that combines convolution and attention mechanisms, achieving competitive results while reducing the number of parameters. It utilizes a pose estimation method based on convolution and multi-head attention modules, which reduces errors compared to other methods and exhibits good robustness. By using only endoscopic images as input, it not only reduces the number of network model parameters while maintaining performance but also improves the inference speed of the depth estimation network.

[0114] Example 2

[0115] like Figure 7As shown, this embodiment of the invention provides a lightweight depth estimation system based on neural networks, comprising the following modules:

[0116] The data preprocessing module is used to convert video data into image sequences, segment the image sequences, perform data cleaning, and data enhancement.

[0117] The depth estimation module is built based on convolution and attention mechanisms. It takes a single image as input and outputs the corresponding 3D information of the image. Specifically, the convolution module is used to process the layered image to obtain local texture features, and the attention mechanism module is used to process the layered image to obtain global contour features, estimating the depth of each pixel in the input image.

[0118] The pose estimation module is built based on convolution and multi-head attention modules. It takes two concatenated adjacent images as input, first uses the convolution module to process the images to obtain image features, and then uses the multi-head attention module to process the feature maps for feature enhancement, and outputs the pose matrix between adjacent images.

[0119] The point cloud fusion module reconstructs point clouds based on camera intrinsic parameters and depth information generated from single images; it then merges multiple point clouds into a single global voxel based on the transformation matrix obtained from pose estimation, and displays the final result.

[0120] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A lightweight depth estimation method based on neural networks, characterized in that, include: Step S1: Data preprocessing, including converting video data into image sequences, segmenting the image sequences, and performing data cleaning and data augmentation; Step S2: Build a depth estimation network based on convolution and attention mechanisms. Take a single image as input and output the depth information corresponding to the image. Among them, layer processing based on convolution operation is used to obtain local texture features, and layer processing based on attention mechanism operation is used to obtain global contour features. Step S3: Construct a pose estimation network based on convolution and multi-head attention modules. Take two cascaded adjacent images as input and output the transformation matrix between adjacent images. First, convolution is used to process the images to obtain image features, and then multi-head attention modules are used to process the image features for feature enhancement. At the same time, a loss function is set to constrain the two neural networks, the depth estimation network and the pose estimation network. Step S4: Recover the point cloud based on the camera's intrinsic parameters and the depth information generated from the single image in Step S2; merge multiple point clouds into a global voxel based on the transformation matrix obtained from the pose estimation network in Step S3, and display the final result.

2. The lightweight depth estimation method based on neural networks according to claim 1, characterized in that, Step S2: Construct a depth estimation network based on convolution and attention mechanisms, taking a single image as input and outputting the depth information corresponding to the image, specifically including: Step S21: Perform pyramid downsampling on the input image I to obtain thumbnails I1, I2, and I3 respectively; I1, I2, and I3 are thumbnails of the original image downsampled at different resolutions. Step S22: The depth estimation network uses four encoders to receive the original images I, I1, I2 and I3 respectively, which are the dimensionality reduction encoder, the low-dimensional texture encoder, the contour encoder and the high-dimensional texture encoder; wherein the dimensionality reduction encoder receives the original image I, the low-dimensional texture encoder receives I1, the contour encoder receives I2 and the high-dimensional texture encoder receives I3. The dimension reduction encoder receives the original image I and encodes the image to obtain a low-dimensional feature map F0. The low-dimensional texture encoder encodes low-dimensional feature maps F0 and I1 to obtain feature map F1. The low-dimensional texture encoder comprises a dimensionality-reducing convolution, two types of convolution modules, and an attention mechanism module with an attention function. The two types of convolution modules are stacked alternately to reduce the number of parameters. The low-dimensional feature maps F0 and I1 are concatenated and input into the dimensionality-reducing convolution to generate a dimensionality-reduced feature S0. S0 then passes through multiple convolution modules and an attention mechanism module with an attention function to generate feature map F1. The contour encoder encodes feature maps F1, S0, and I2 to obtain feature map F2. The contour encoder consists of two dimensionality-reducing convolutions, multiple attention modules with pooling, and two attention mechanism modules with attention functions. Features F1, S0, and I2 are concatenated and input into the dimensionality-reducing convolution to generate dimensionality-reduced feature S1. S1 is then passed through multiple attention mechanism modules with pooling, another dimensionality-reducing convolution, and two attention mechanism modules with attention functions to generate feature map F2. The high-dimensional texture encoder encodes feature maps F2, S1, and I3 to obtain feature map F3. The high-dimensional texture encoder consists of dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function. Two types of convolutional modules are stacked alternately in the high-dimensional texture encoder to reduce the number of parameters. Feature maps F2, S1, and I3 are concatenated, undergoing dimensionality-reducing convolution, multiple convolutional modules, and an attention mechanism module with an attention function to generate feature map F3. Step S23: The depth estimation network uses three decoders to receive the outputs of encoders of different dimensions, namely a high-dimensional decoder, a medium-dimensional decoder, and a low-dimensional decoder. The high-dimensional decoder receives the output F3 and output U3 of the high-dimensional texture encoder, and predicts the low-resolution depth information D3. The mid-dimensional decoder receives the output F2 of the contour encoder and the output U3 of the high-dimensional decoder, outputs U2, and predicts the mid-resolution depth information D2. The low-dimensional decoder receives the output F1 of the low-dimensional texture encoder and the output U2 of the mid-dimensional decoder, outputs feature U1, and predicts high-resolution depth information D1.

3. The lightweight depth estimation method based on neural networks according to claim 2, characterized in that, A pose estimation network is built based on convolutional and multi-head attention modules. It takes two concatenated adjacent images as input and outputs a transformation matrix between adjacent images. First, a convolutional module is used to process the images to obtain image features, and then a multi-head attention module is used to process these features for feature enhancement. Simultaneously, a loss function is set to constrain both the pose estimation network and the depth estimation network, specifically including: Step S31: Assume the original image is I k , the original image I k and adjacent image I k+1 Cascaded to form cascaded input I p ; Step S32: The pose estimation network uses three feature extractors to process the cascaded input I. p Layer-by-layer feature extraction and enhancement are performed, namely, low-dimensional feature extraction enhancer, high-dimensional feature extraction enhancer, and high-dimensional feature extractor; Among them, the high-dimensional feature extractor consists of convolution operations, while both the low-dimensional feature extraction enhancer and the high-dimensional feature extraction enhancer consist of convolution operations and multi-head attention operations. The low-dimensional feature extraction enhancer receives input I. p Encode the cascaded images to output low-dimensional features P1; The high-dimensional feature extraction enhancer receives low-dimensional feature P1, encodes P1, and extracts high-dimensional feature P2. The high-dimensional feature extractor receives high-dimensional feature P2, encodes P2, and outputs pose feature P; Step S33: Generate a vector by convolution based on the pose feature P, and then convert it into a transformation matrix; Step S34: Based on the structure and objective of the pose estimation network and depth estimation network, define the loss function, including a similarity loss based on photometric correction. and edge smoothing loss Among them, similarity loss based on photometric correction The similarity loss is calculated using a structural similarity index, as shown in formula (1): Wherein, SSIM is the structural similarity index calculation function. It is an image that has been photometrically corrected. It is an image back-projected based on predicted depth information, camera parameters, and predicted pose; α is set to 0.85; The edge smoothing loss As shown in formula (2): Where d is the predicted depth, It is the gradient of d in the x-direction. It is the gradient of d in the y direction. It is the gradient of I in the x-direction. is the gradient of I in the y-direction, where I is the input image.

4. The lightweight depth estimation method based on neural networks according to claim 3, characterized in that, Step S4: Recover the point cloud based on the camera's intrinsic parameters and the depth information generated from a single image; merge multiple point clouds into a global point cloud based on the transformation matrix obtained from pose estimation; and display the final result. Specifically, this includes: The method for recovering point clouds based on camera intrinsic parameters and depth information generated from a single image is shown in the formula: Among them, (uv 1) T These are homogenized image coordinates, (xyz). T It represents the recovered 3D coordinates, d is the depth estimated by the network, and f is the depth. x and f y It refers to the camera's horizontal and vertical focal lengths, c x and c y This represents the offset of the camera's optical axis in the image coordinate system; Based on the series of transformation matrices M1, ..., M2 obtained from the pose estimation network n Multiple point clouds are fused using a truncated signed distance function (TSDF), where the depth map D is used as the fusion point. t and transformation matrix M t The TSDF value and weight of the current frame are respectively represented by g t and w t This indicates that the TSDF values ​​and weights of the global voxels before the update are represented by G. t-1 and W t-1 This indicates that the updated global voxel TSDF values ​​and weights are represented by G. t and W t This means that multiple depths are fused into a single global voxel, as shown in the formula: IN t =In t-1 +in t 。 5. A lightweight depth estimation system based on neural networks, characterized in that, Includes the following modules: The data preprocessing module is used to convert video data into image sequences, segment the image sequences, perform data cleaning, and data enhancement. The depth estimation module is built based on convolution and attention mechanisms. It takes a single image as input and outputs the depth information of the image. Specifically, the convolution module is used to process the layered image to obtain local texture features, and the attention mechanism module is used to process the layered image to obtain global contour features, thus estimating the depth of each pixel in the input image. The pose estimation module is built based on convolution and multi-head attention modules. It takes two concatenated adjacent images as input, first uses the convolution module to process the images to obtain image features, and then uses the multi-head attention module to process the image features for feature enhancement, and outputs the transformation matrix between adjacent images. The point cloud fusion module reconstructs point clouds based on camera intrinsic parameters and depth information generated from single images; it then merges multiple point clouds into a single global voxel based on the transformation matrix obtained from pose estimation, and displays the final result.

Images