A Depth Estimation Method and System Based on Variable Convolution Inter-Frame Pixel Matching

Abstract

This invention discloses a depth estimation method and system based on variable convolutional inter-frame pixel matching, belonging to the field of depth estimation technology. The method includes: inputting a target frame image and a source frame image into a pose prediction network to obtain the camera's pose changes; concatenating the target frame image and the source frame image and inputting the result into an inter-frame offset learning network to obtain inter-frame pixel matching relationships; reconstructing the target frame image using the inter-frame pixel matching relationships through a variable convolutional network, using the reconstruction error of the target frame image as a supervision signal to learn inter-frame offsets during the reconstruction process; converting the inter-frame offsets into sequential depth information based on camera intrinsic parameters and camera pose changes through the inter-frame pixel matching relationships; and inputting the target frame image and the sequential depth information in parallel into a depth estimation network to obtain the depth map corresponding to the target frame image. The use of variable convolution achieves adaptive matching under scale changes, improving the accuracy of depth estimation and solving the problem of depth estimation accuracy being affected by scale changes.

Description

Technical Field

[0001] This invention relates to the field of depth estimation technology, and in particular to a depth estimation method and system based on variable convolutional inter-frame pixel matching. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] In recent years, with the rise of artificial intelligence, autonomous driving technology has received increasing attention. Depth estimation is a crucial step in the autonomous driving process; accurate depth information is essential for accurately describing the vehicle's location and the depth of objects, thus enabling a series of autonomous driving maneuvers.

[0004] Currently, depth estimation in autonomous driving mainly relies on scene acquisition using color cameras and depth acquisition using radar sensors. In terms of depth estimation based on color cameras, most methods are based on viewpoint domain disparity (parallax). However, these methods require a relatively dense camera setup, which places high demands on hardware and computation in wide-area scenarios, failing to meet the requirements of low cost and low power consumption in autonomous driving. On the other hand, radar sensors cannot acquire dense scene depth maps and are relatively expensive, making it difficult to independently generate high-quality complete scene depth maps.

[0005] In the search for efficient scene depth estimation methods, monocular video-based depth estimation methods have gradually emerged and attracted widespread attention from academia and industry. Monocular video-based depth estimation methods can provide an unsupervised learning approach, utilizing the relationships between different frames of the video for supervision, eliminating the need for depth labels and greatly simplifying the preparation of training samples.

[0006] However, the current mainstream monocular depth estimation method is based on a single frame texture map and uses deep learning technology to estimate the depth map. Since there is no explicit geometric depth information in this process (such as disparity matching in multi-view images), the model tends to develop depth cues in the image, such as the blurring degree of objects in the scene and occlusion relationships. However, since there is no clear scene geometric information in this process, its accuracy is difficult to guarantee and cannot meet the high requirements of autonomous driving. Therefore, it is necessary to make full use of the correspondence between different frames to estimate scene geometric information.

[0007] However, compared to multi-view images / videos, monocular videos under autonomous driving are acquired during vehicle movement, and there are a lot of scale scaling problems between consecutive frames. That is, the same target has different scales in different frames, resulting in low matching efficiency between different frames, and traditional matching algorithms are difficult to apply directly. Moreover, traditional matching algorithms do not consider the motion information of consecutive frames in monocular videos, affecting the accuracy of the final depth estimation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a depth estimation method, system, electronic device, and computer-readable storage medium based on variable convolutional inter-frame pixel matching. By leveraging variable convolutional networks, the inter-frame pixel matching relationships containing depth information of video sequences can be effectively learned, improving the matching efficiency and accuracy between different frame images, thereby enhancing the accuracy and efficiency of depth estimation.

[0009] In a first aspect, the present invention provides a depth estimation method based on variable convolutional inter-frame pixel matching;

[0010] A depth estimation method based on variable convolutional inter-frame pixel matching includes:

[0011] Acquire the target frame image and the source frame image, and input the target frame image and the source frame image into the trained pose prediction network to obtain the camera pose change;

[0012] The target frame image and the source frame image are concatenated and then input into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship.

[0013] The variable convolutional network reconstructs the target frame image through inter-frame pixel matching. During the reconstruction process, the reconstruction error of the target frame image is used as a supervision signal to learn the inter-frame offset. Based on the camera intrinsic parameters and the camera pose change, the inter-frame offset is converted into sequence depth information through inter-frame pixel matching.

[0014] The target frame image and sequence depth information are input into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image.

[0015] Furthermore, the step of concatenating the target frame image and the source frame image and inputting the result into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship specifically involves:

[0016] The target frame image and the source frame image are stitched together and then input into an inter-frame offset learning network based on the U-Net structure for processing to obtain the inter-frame pixel matching relationship.

[0017] The U-Net structure is based on a fully convolutional network, using the output of the convolutional layer as the input of the corresponding deconvolutional layer.

[0018] Furthermore, the step of inputting the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image specifically includes:

[0019] The sequence depth information is input into the residual network for feature extraction to obtain the first depth feature at multiple scales; the target frame image is input into the residual network for feature extraction to obtain the second depth feature at multiple scales.

[0020] The first and second depth features are stitched together according to scale, and then input into the depth encoder step by step according to the scale level to obtain the depth feature fusion map.

[0021] The depth feature fusion map is input into the depth decoder for step-by-step decoding, and a depth map with the same scale as the target frame image is obtained by linear interpolation.

[0022] Furthermore, based on camera intrinsic parameters and camera pose changes, the inter-frame offset is converted into sequence depth information through inter-frame pixel matching relationships as follows:

[0023] <KT t→s K -1 D0>I t -I t = offset(s→t),

[0024] Where K is the camera intrinsic parameter, T t→s For camera pose changes, D0 represents sequence depth information, I t The target frame image is denoted as offset(s→t), which represents the inter-frame offset between the source frame image and the target frame image.

[0025] Furthermore, the variable convolutional network dynamically adjusts the sampling position of the convolutional kernels by using the offset in the inter-frame pixel matching relationship.

[0026] Furthermore, the step of inputting the target frame image and the source frame image into the trained pose prediction network to obtain the camera's changed pose specifically involves: concatenating the target frame image and the source frame image and inputting them into the residual network for processing to extract camera pose features; then inputting the camera pose features into the convolutional layer to estimate the camera's changed pose.

[0027] Furthermore, the loss function for training the variable convolutional network is expressed as:

[0028]

[0029] Where a is a constant, SSIM is the structural similarity, and I s→t For the target frame image to be reconstructed, I t The target frame image.

[0030] Secondly, the present invention provides a depth estimation system based on variable convolutional inter-frame pixel matching;

[0031] A depth estimation system based on variable convolutional inter-frame pixel matching includes:

[0032] The pose prediction module is configured to: acquire target frame images and source frame images, input the target frame images and source frame images into the trained pose prediction network, and acquire the camera's changing pose;

[0033] The inter-frame pixel matching relationship generation module is configured to: stitch the target frame image and the source frame image together and input them into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship; the variable convolutional network reconstructs the target frame image through the inter-frame pixel matching relationship, and during the reconstruction process, the reconstruction error of the target frame image is used as a supervision signal to learn the inter-frame offset; according to the camera intrinsic parameters and the camera pose change, the inter-frame offset is converted into sequence depth information through the inter-frame pixel matching relationship.

[0034] The depth estimation module is configured to input the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image.

[0035] Thirdly, the present invention provides an electronic device;

[0036] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the depth estimation method based on variable convolutional inter-frame pixel matching described above.

[0037] Fourthly, the present invention provides a computer-readable storage medium;

[0038] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps of the aforementioned depth estimation method based on variable convolutional inter-frame pixel matching.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] 1. The technical solution provided by this invention can effectively learn the inter-frame pixel matching relationship containing the depth information of video sequences by means of a variable convolutional network; the size and position of the deformable convolutional kernel in the deformable convolutional network can be dynamically adjusted according to the image content that needs to be identified. Its intuitive effect is that the sampling point position of the convolutional kernel at different positions will change adaptively according to the image content, thereby adapting to the geometric deformation such as the shape and size of different objects.

[0041] 2. The technical solution provided by this invention performs self-supervised multi-frame joint depth estimation based on inter-frame depth information. By generating inter-frame pixel matching relationships, the matching efficiency between different frames is improved, thereby improving the efficiency of depth estimation.

[0042] 3. The technical solution provided by the present invention obtains the motion changes of objects in continuous frames by using deformable convolution and the acquisition of offsets in deformable convolution. This motion change information is added to the encoding and decoding process of depth estimation, and the added offset information makes the depth encoding and decoding information richer, ultimately improving the accuracy of depth estimation. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0044] Figure 1 A schematic diagram illustrating the process of learning inter-frame pixel matching relationships based on variable convolution provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a depth estimation network architecture based on variable convolutional inter-frame pixel matching provided in an embodiment of the present invention. Detailed Implementation

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0049] Example 1

[0050] Existing monocular depth estimation methods lack explicit scene geometry information, which cannot guarantee the accuracy of depth estimation. Furthermore, monocular videos are acquired during vehicle movement, resulting in significant scale differences between consecutive frames and low matching efficiency between different frames. Therefore, this invention provides a depth estimation method based on variable convolutional inter-frame pixel matching.

[0051] Next, combined Figures 1-2 This embodiment discloses a depth estimation method based on variable convolutional inter-frame pixel matching, which includes the following steps:

[0052] S1. Obtain the target frame image and the source frame image, wherein the target frame image and the source frame image are adjacent frame images, extracted from the monocular video sequence collected during the autonomous driving process.

[0053] For example, taking three frames from a single video sequence as an example, it includes the target frame image I. t and the two adjacent source frame images I s , s∈{-1,1}.

[0054] S2. Input the target frame image and the source frame image into the trained pose prediction network to obtain the camera's pose changes.

[0055] The pose prediction network consists of a residual network and three cascaded convolutional layers.

[0056] Specifically, the target frame and the source frame are first stitched together and then fed into the residual network for learning. After processing through convolutional layers, residual blocks, fully connected layers and other network layers, camera pose features are extracted. The camera pose features are then fed into three cascaded convolutional layers. Through processing by the three cascaded convolutional layers, the final camera pose change is obtained.

[0057] S3. After concatenating the target frame image and the source frame image, input them into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship. Here, the inter-frame pixel matching relationship refers to the initial offset.

[0058] The initial offset and the source frame image are input into the variable convolutional network. The variable convolutional network reconstructs the target frame image through inter-frame pixel matching. During the reconstruction of the target frame image, the inter-frame offset is learned. Based on the camera intrinsic parameters and the camera pose change, the inter-frame offset is converted into sequence depth information through inter-frame pixel matching.

[0059] The modular geometry used in traditional convolutional networks is fixed, and their ability to model geometric transformations is inherently limited. In other words, the regular grid sampling of standard convolution makes it difficult for the network to adapt to geometric deformations.

[0060] In monocular video, the scale of objects changes due to camera movement between adjacent frames, making it difficult to directly match pixels between adjacent video frames in multi-frame joint depth estimation. However, variable convolutional networks can automatically adjust the size of the network's receptive field by further shifting the spatial sampling position information, thereby enabling matching of objects at different scales. Therefore, in this embodiment, a variable convolutional network is used to learn the inter-frame pixel matching relationship containing the depth information of the video sequence.

[0061] Deformable convolutional networks add an offset to each position of the convolutional kernel. These offsets allow the kernel to sample arbitrarily at the current position, no longer limited to the previous regular grid points. In fact, the offsets added in deformable convolutional networks are part of the network structure, calculated through another parallel standard convolutional unit, and can then be used for end-to-end learning via gradient backpropagation.

[0062] After learning this offset, the size and position of the deformable convolutional kernel can be dynamically adjusted according to the image content to be recognized. The intuitive effect is that the sampling point position of the convolutional kernel at different locations will adaptively change according to the image content, thereby adapting to the geometric deformations such as the shape and size of different objects. The network first obtains the displacement required for variable convolution using the output of a convolutional layer, and then applies it to the convolutional kernel to achieve the effect of variable convolution.

[0063] For example, in combination Figure 1 The specific process of step S3 is as follows:

[0064] S301, Transfer the source frame image I -1 and target frame image I t and source frame image I1 and target frame image I t The two sets of images, stitched together, are sequentially input into an inter-frame offset learning network based on a U-Net structure with full convolution. The output dimension of the last layer of the network is controlled to be (H, W, 18) to obtain the inter-frame pixel matching relationship. Here, H and W are the height and width of the original input image, and 18 represents the offset of the 9 positions in the 3*3 convolution kernel at positions H and W in the image. In this embodiment, the inter-frame pixel matching relationship refers to the initial offset.

[0065] Unlike classic convolutional neural networks that use fully connected layers after convolutional layers to obtain fixed-length feature vectors, fully convolutional networks can accept input images of any scale and use deconvolutional layers to upsample the feature maps of the last convolutional layer, restoring them to the same scale as the input image. This allows for a prediction for each pixel while preserving the spatial information of the original input image. The U-Net structure, building upon fully convolutional networks, uses the output of convolutional layers as the input to the corresponding deconvolutional layers, enhancing the deconvolution effect and obtaining more detailed information.

[0066] Step 302: Input the initial offset and the source frame image into the variable convolutional network. Add an offset to each position of the convolution kernel in the variable convolutional network. The variable convolutional network then processes the source frame image to reconstruct the target frame image, that is, to map the source frame image onto the color image of the target frame image, thereby providing a supervision signal for learning the inter-frame pixel matching relationship, and finally learning the optimal inter-frame offset.

[0067] Network training is supervised using a combination of L1 and SSIM loss functions, as shown in the following equation:

[0068]

[0069] In the formula, a is a constant, which is set to 0.85 in this embodiment, SSIM is the structural similarity, and I s→t For the target frame image to be reconstructed, I t The target frame image.

[0070] Step 303: Convert the inter-frame offset obtained based on variable convolution into the depth information of the target frame to better facilitate the learning of the neural network.

[0071] The pixel matching relationship from the source frame to the target frame represents the positional change of the same target pixel from the source frame to the target frame due to camera motion. The inter-frame pixel matching relationship (H,W,18) is divided into 9 groups (H,W,2), as shown in the following formula. For the position coordinates of each target frame, the camera intrinsic parameter K and the camera pose T predicted by the self-supervised depth estimation network are used. t→s Convert the depth information (H, W, 9) of the target frame.

[0072] <KT t→s K -1 D0>I t -I t = offset(s→t)

[0073] Where K is the camera intrinsic parameter, T t→s For camera pose changes, D0 represents sequence depth information, I t The target frame image is denoted as offset(s→t), which represents the inter-frame offset between the source frame image and the target frame image.

[0074] S4. Input the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image.

[0075] Specifically, the sequence depth information obtained from the inter-frame pixel matching relationship is used to obtain multi-scale depth features using a residual network. The target frame image is also used to obtain multi-scale depth features using a residual network, which are then stitched together according to scale and fed into the depth encoder step by step to complete feature fusion based on multi-frame depth information.

[0076] Then, the depth decoder decodes the fused depth features, decodes them step by step to obtain depth maps at four scales, and then linearly interpolates them to a depth map at the same scale as the input image to complete the reconstruction of the target frame at the scale of the input image.

[0077] As shown in the following formula, for each pixel position of the target frame, through the depth map D t Video source frame I s With video target frame I t Camera pose change T t→s Given the camera intrinsic parameter K, calculate the corresponding source frame position coordinates, then assign the obtained source frame position pixel value to the corresponding target frame position pixel value to reconstruct the target frame, and use the target frame reconstruction error to construct a supervision signal.

[0078] I s→t =I s <proj(D t ,T t→s ,K)>

[0079] For target frame reconstruction errors from depth maps of different source frames and different scales, the minimum reconstruction error, rather than the average reconstruction error of multiple source frames, is used as the final optical reconstruction loss function, as follows:

[0080]

[0081] This is because when some problematic pixels only appear in the target frame and not in the source frame, even if the network can accurately predict their depth, they cannot be matched with the corresponding source frame pixels due to occlusion, resulting in a large reprojection error penalty.

[0082] In addition to the optical reconstruction loss function, an edge smoothing function is also used to optimize the depth map prediction performance, as follows:

[0083]

[0084] Furthermore, self-supervised monocular depth estimation networks are typically trained under the assumption that the camera is moving and the scene within the frame is static. When this assumption is broken, such as when the camera is fixed or there are moving objects in the scene, the network's depth prediction performance is significantly affected. In video sequences, pixels that remain identical in adjacent frames often represent a stationary camera, moving objects, or a low-texture region. By setting a simple binary automask u, u is set to 1 only when the reconstruction error between the target frame and the source frame is less than the reconstruction error between the target frame and the source frame, and the network loss function includes the optical reconstruction loss function. This effectively filters out pixels that remain unchanged from one frame to the next in the video. The effect of this is to make the network ignore objects moving at the same speed as the camera, and even ignore entire frames in the monocular video when the camera is stationary.

[0085] The final overall loss function for network training is:

[0086] L = uL P +L S .

[0087] The depth estimation method based on variable convolutional inter-frame pixel matching described in this embodiment significantly improves the depth estimation performance. To further verify the effectiveness of this method, it was applied to experiments on the KITTI dataset on Monodepth2, and it can be seen that the experimental results using this method are significantly improved.

[0088] Table 1 Comparison of experimental results on the KITTI dataset

[0089]

[0090]

[0091] Example 2

[0092] This embodiment discloses a depth estimation system based on variable convolutional inter-frame pixel matching, including:

[0093] The pose prediction module is configured to: acquire target frame images and source frame images, input the target frame images and source frame images into the trained pose prediction network, and acquire the camera's changing pose;

[0094] The inter-frame pixel matching relationship generation module is configured to: stitch the target frame image and the source frame image together and input them into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship; the variable convolutional network reconstructs the target frame image through the inter-frame pixel matching relationship, and during the reconstruction process, the reconstruction error of the target frame image is used as a supervision signal to learn the inter-frame offset; according to the camera intrinsic parameters and the camera pose change, the position coordinates of the reconstructed target frame image are converted into sequence depth information through the inter-frame pixel matching relationship;

[0095] The depth estimation module is configured to input the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image.

[0096] It should be noted that the pose prediction module, inter-frame pixel matching relationship generation module, and depth estimation module described above correspond to the steps in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0097] Example 3

[0098] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the depth estimation method based on variable convolutional inter-frame pixel matching described above.

[0099] Example 4

[0100] Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the depth estimation method based on variable convolutional inter-frame pixel matching described above.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A depth estimation method based on variable convolutional inter-frame pixel matching, characterized in that, include: Acquire the target frame image and the source frame image, and input the target frame image and the source frame image into the trained pose prediction network to obtain the camera pose change; The target frame image and the source frame image are concatenated and then input into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship. The variable convolutional network reconstructs the target frame image through inter-frame pixel matching. During the reconstruction process, the reconstruction error of the target frame image is used as a supervision signal to learn the inter-frame offset. Obtain camera intrinsic parameters, and based on the camera intrinsic parameters and camera pose changes, convert inter-frame offsets into sequence depth information through inter-frame pixel matching relationships; The target frame image and sequence depth information are input into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image; The specific steps for concatenating the target frame image and the source frame image and inputting them into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship are as follows: The target frame image and the source frame image are stitched together and then input into an inter-frame offset learning network based on the U-Net structure for processing to obtain the inter-frame pixel matching relationship. The U-Net structure is based on a fully convolutional network, using the output of the convolutional layer as the input of the corresponding deconvolutional layer. The step of inputting the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image specifically includes: The sequence depth information is input into the residual network for feature extraction to obtain the first depth feature at multiple scales; the target frame image is input into the residual network for feature extraction to obtain the second depth feature at multiple scales. The first and second depth features are stitched together according to scale, and then input into the depth encoder step by step according to the scale level to obtain the depth feature fusion map. The depth feature fusion map is input into the depth decoder for step-by-step decoding, and a depth map with the same scale as the target frame image is obtained by linear interpolation. Based on camera intrinsic parameters and camera pose changes, the inter-frame offset is converted into sequence depth information through inter-frame pixel matching relationships, as shown below: ， in, For camera internal parameters, Change the camera's pose. For sequence depth information, For the target frame image, This represents the inter-frame offset between the source frame image and the target frame image.

2. The depth estimation method based on variable convolutional inter-frame pixel matching as described in claim 1, characterized in that, The variable convolutional network dynamically adjusts the sampling position of the convolutional kernel by using the offset in the inter-frame pixel matching relationship.

3. The depth estimation method based on variable convolutional inter-frame pixel matching as described in claim 1, characterized in that, The step of inputting the target frame image and the source frame image into the trained pose prediction network to obtain the camera pose change is as follows: the target frame image and the source frame image are concatenated and then input into the residual network for processing to extract camera pose features; the camera pose features are input into the convolutional layer to estimate the camera pose change.

4. The depth estimation method based on variable convolutional inter-frame pixel matching as described in claim 1, characterized in that, The loss function for training the variable convolutional network is expressed as: ， in, It is a constant. For structural similarity, For the reconstructed target frame image, The target frame image.

5. A depth estimation system based on variable convolutional inter-frame pixel matching, using the method described in any one of claims 1-4, characterized in that, include: The pose prediction module is configured to: acquire target frame images and source frame images, input the target frame images and source frame images into the trained pose prediction network, and acquire the camera's changing pose; The inter-frame pixel matching relationship generation module is configured to: stitch the target frame image and the source frame image together and input them into the trained inter-frame offset learning network to obtain the inter-frame pixel matching relationship; the variable convolutional network reconstructs the target frame image through the inter-frame pixel matching relationship, and during the reconstruction process, the reconstruction error of the target frame image is used as a supervision signal to learn the inter-frame offset; according to the camera intrinsic parameters and the camera pose change, the inter-frame offset is converted into sequence depth information through the inter-frame pixel matching relationship. The depth estimation module is configured to input the target frame image and sequence depth information into the depth estimation network in parallel to obtain the depth map corresponding to the target frame image.

6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the depth estimation method based on variable convolutional inter-frame pixel matching as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the depth estimation method based on variable convolutional inter-frame pixel matching as described in any one of claims 1-4.

Images