Image depth estimation method, device, equipment, medium and computer program product

Abstract

The invention relates to the field of image processing, and provides an image depth estimation method, device and equipment, a medium and a computer program product. Inputting left and right eye images including the left eye adjacent frame image and the corrected binocular image into a depth estimation model including a spatio-temporal information extraction network, a binocular stereo matching network and a depth evaluation network to obtain a depth map; the spatio-temporal information extraction network is used for performing spatio-temporal feature extraction on adjacent frames of images of a left eye to obtain binocular pose transformation information and image change information between the adjacent frames of images; the binocular stereo matching network is used for performing multi-layer feature extraction on the corrected binocular image to obtain a multi-level disparity map; and the depth evaluation network is used for evaluating the binocular camera pose transformation information, the image target change information and the multi-level disparity map to obtain a depth map of the left-eye image and the right-eye image. According to the method, the precision of image depth evaluation is improved by learning the dynamic change rule between binocular image frames and utilizing motion information and time sequence consistency in a dynamic scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image depth estimation method, device, equipment, medium and computer program product. BACKGROUND

[0002] With the development of automation technology, the grabbing robot has been widely used in automatic production scenes (for example, article carrying and part installation, etc.). Then, the light environment in the natural production scene is relatively complex, and the existing scheme for binocular depth estimation only considers the stereo matching of a single frame image. This limitation leads to the problem of low depth estimation accuracy of the existing scheme for binocular depth estimation. SUMMARY

[0003] The present application provides an image depth estimation method, device, equipment, medium and computer program product, to solve the problem of low depth estimation accuracy of the existing scheme for binocular depth estimation, and to achieve the effect of improving the image depth estimation accuracy.

[0004] The present application provides an image depth estimation method, comprising the following steps: inputting left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images include a left eye adjacent frame image and a corrected binocular image; The depth estimation model includes a space-time information extraction network, a binocular stereo matching network and a depth evaluation network. The space-time information extraction network includes a plurality of skip-connection codec modules, and each codec module is configured to: extracting space-time features from the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; The binocular stereo matching network is configured to extract multi-layer features from the corrected binocular image to obtain multi-level disparity maps. The depth evaluation network is configured to evaluate the binocular camera pose transformation information, the image target change information and the multi-level disparity maps to obtain the depth map of the left and right eye images.

[0005] According to the image depth estimation method provided by the present application, the codec module includes an encoder, a spatial attention network and a decoder. The encoder is configured to obtain initial encoding features of the left eye adjacent frame image. The spatial attention network is configured to extract texture features, edge perception enhancement and anti-light area suppression from the initial encoding features to obtain multi-scale attention features. The decoder is configured to decode the multi-scale attention feature to obtain binocular camera pose transformation information and image target change information between adjacent frame images.

[0006] According to the image depth estimation method provided by the application, the codec module comprises a pose transformation codec, an optical flow codec and an appearance flow codec; the binocular camera pose transformation information is a binocular pose transformation matrix; the image target change information comprises an optical flow map and an appearance flow map; wherein: The decoder of the pose transformation codec is configured to perform dimension compression processing and feature refining processing on the feature output by the encoder of the pose transformation codec to obtain the binocular pose transformation matrix. The decoder of the optical flow codec is configured to decode the multi-scale attention feature output by the spatial attention network of the optical flow codec to obtain the optical flow map. The appearance flow codec is configured to extract features from the left adjacent frame image and the optical flow map to obtain the appearance flow map.

[0007] According to the image depth estimation method provided by the application, the spatial attention network comprises a texture feature extraction network, an edge perception enhancement network, an anti-reflection region suppression network, a channel splicing module and a residual connection network, wherein: The texture feature extraction network is configured to extract texture features from the initial encoding feature to obtain image texture features. The edge perception enhancement network is configured to extract edge features from the initial encoding feature to obtain image edge features. The anti-reflection region suppression network is configured to extract anti-reflection suppression features from the initial encoding feature to obtain image anti-reflection suppression features. The channel splicing module is configured to perform channel dimension splicing processing on the image texture features, the image edge features and the image anti-reflection suppression features to obtain an attention map. The residual connection network is configured to perform residual connection on the attention map and the initial encoding feature to obtain a multi-scale attention feature.

[0008] According to the image depth estimation method provided by the application, the feature extraction from the left adjacent frame image and the optical flow map to obtain the appearance flow map comprises: Obtaining the brightness change value of each pixel in the left adjacent frame image; Determining the luminosity change value of each pixel based on the brightness change value and a geometric deformation field; the geometric deformation field is determined based on the optical flow map; Determining the appearance flow map based on the luminosity change value of each pixel.

[0009] According to the image depth estimation method provided by the application, the binocular stereo matching network comprises a multi-layer feature extraction module, a stereo matching module and a self-attention module, wherein: The multi-layer feature extraction module is configured to extract multi-layer features of the corrected binocular image. The stereo matching module is configured to determine a feature similarity matrix based on the multi-layer features. The self-attention module is configured to perform regional propagation processing on a multi-level disparity map to obtain a multi-stage disparity map; the multi-level disparity map is generated based on the feature similarity matrix.

[0010] The application further provides an image depth estimation device comprising the following modules: The image depth estimation module is configured to input left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images comprise a left eye adjacent frame image and a corrected binocular image. The depth estimation model comprises a space-time information extraction network, a binocular stereo matching network and a depth estimation network. The space-time information extraction network comprises a plurality of skip-connection codec modules, and each codec module is configured to: extract space-time features of the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images. The binocular stereo matching network is configured to extract multi-layer features of the corrected binocular image to obtain a multi-stage disparity map. The depth estimation network is configured to estimate the binocular camera pose transformation information, the image target change information and the multi-stage disparity map to obtain a depth map of the left and right eye images.

[0011] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the image depth estimation method of any of the above when executing the computer program.

[0012] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the image depth estimation method of any of the above.

[0013] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the image depth estimation method of any of the above.

[0014] The image depth estimation method, device, equipment, medium and computer program product provided by the application, through a plurality of jump connection codec modules, the spatio-temporal features of the left eye adjacent frame image are extracted to obtain the spatio-temporal information between adjacent frame images including the binocular camera pose transformation information and the image target change information; again, through extracting the multi-layer features of the corrected binocular image, using the structural similarity between the left eye image and the right eye image, the accurate prediction result of the reliable area of the image is propagated to other areas to obtain a multi-level disparity map. Through the depth evaluation network, the spatio-temporal dynamic information learned by the spatio-temporal information extraction network is used to evaluate the multi-level disparity map to obtain the depth map of the left eye image and the right eye image. The application learns the dynamic change rule between the binocular image frames, uses the motion information and the time sequence consistency in the dynamic scene, and improves the accuracy of image depth evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a flowchart of the image depth estimation method provided by the application.

[0017] Figure 2 is a network framework schematic diagram of the depth estimation model provided by the application.

[0018] Figure 3 is a framework schematic diagram of the spatial attention network provided by the application.

[0019] Figure 4 is a structural schematic diagram of the image depth estimation device provided by the application.

[0020] Figure 5 is a structural schematic diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the application more clear, the technical scheme in the application will be described clearly and completely in the following combined with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0022] The image depth estimation method, device, equipment, medium and computer program product of the application will be described in the following combined with Figures 1-5 the drawings.

[0023] Figure 1 is one of the flowcharts of the image depth estimation method provided by the present application, as Figure 1 shown, the method comprises the following: Step 100, input left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images include a left eye adjacent frame image and a corrected binocular image; Wherein, the depth estimation model comprises a space-time information extraction network, a binocular stereo matching network and a depth evaluation network; The space-time information extraction network comprises a plurality of skip-connection codec modules, and each codec module is used for: extracting space-time features from the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; The binocular stereo matching network is used for multi-layer feature extraction on the corrected binocular image to obtain a multi-level disparity map; The depth evaluation network is used for evaluating the binocular camera pose transformation information, the image target change information and the multi-level disparity map to obtain a depth map of the left and right eye images.

[0024] The image depth estimation method provided by the present application realizes the depth information estimation of left and right eye images by training a depth estimation model to obtain a depth map of left and right eye images (a single-channel image recording the distance information of each pixel point in the application scene to the camera).

[0025] As Figure 2 shown, the depth estimation model provided by the present application at least comprises a space-time information extraction network, a binocular stereo matching network and a depth evaluation network, wherein the space-time information extraction network is composed of a plurality of skip-connection codec (i.e. codec module). Taking three codec modules as an example, input the left eye adjacent frame image into the space-time information extraction network, calculate and output the binocular camera pose transformation information (used to describe the relative pose relationship between the left and right eye camera coordinate systems) between adjacent frame images through one codec module in the space-time information extraction network.

[0026] Through the other two codec modules in the space-time information extraction network, the target change information (i.e. image target change information) between adjacent frame images is calculated and output, and the image target change information reflects the dynamic change of pixels between adjacent frame images and the difference of the target between adjacent frame images.

[0027] In the depth estimation, in order to eliminate the light difference (for example, brightness and contrast) between adjacent frame images, avoid the depth estimation model from being disturbed by non-geometric information, construct a loss function against photometric inconsistency, and ensure that the depth estimation model focuses on geometric relationship (for example, disparity and motion) rather than light change. Then, the right eye image is warped to the left eye to generate a pseudo left eye image, the previous frame is warped to the current frame to generate a pseudo current frame image, the warped composite image (pseudo left eye image and pseudo current frame image) is used as a self-supervised signal, the image dynamic change rule (implicit) learned by the spatio-temporal information extraction network is distilled to the depth evaluation network, and the depth evaluation network is used to evaluate the binocular camera pose transformation information, image target change information and multi-level disparity map, so as to obtain the depth map of the left and right eye images.

[0028] In the embodiment, the spatio-temporal feature of the left eye adjacent frame image is extracted by the plurality of jump connection codec modules, so as to obtain the spatio-temporal information including the binocular camera pose transformation information and the image target change information between the adjacent frame images; the multi-level feature of the corrected binocular image is extracted, the structural similarity between the left and right eye images is used, the accurate prediction result of the reliable area of the image is propagated to other areas, and the multi-level disparity map is obtained. The spatio-temporal dynamic information learned by the spatio-temporal information extraction network is evaluated by the depth evaluation network, and the multi-level disparity map is evaluated to obtain the depth map of the left and right eye images. The dynamic change rule between the binocular image frames is learned, the motion information and the time sequence consistency in the dynamic scene are used, and the accuracy of the image depth evaluation is improved.

[0029] In one embodiment, the codec module includes an encoder, a spatial attention network and a decoder. The encoder is configured to acquire initial encoding features of the left eye adjacent frame image. The spatial attention network is configured to perform texture feature extraction, edge perception enhancement and anti-light area suppression on the initial encoding features to obtain multi-scale attention features. The decoder is configured to decode the multi-scale attention features to obtain binocular camera pose transformation information and image target change information between adjacent frame images.

[0030] Specifically, as Figure 2As shown, the space-time information extraction network is composed of multiple hop-connected codec modules, wherein the encoder can be a standard ResNet18 (a lightweight model in a residual network, which solves the degradation problem in the training of a deep convolutional neural network (CNN) by introducing a residual block and a skip connection), and the input channels of ResNet18 are expanded to N x 3 (N is the number of adjacent frame images of the input, which can be set to 2), and the weights of the first layer convolution kernel of ResNet18 are initialized by weighted average to ensure effective fusion of time series information. The spatial attention mechanism (i.e., spatial attention network) is added behind ResNet18, which can better extract the features of adjacent frame images.

[0031] The initial encoding features output by the encoder and the three levels of features (texture features, edge features, and anti-reflection features) extracted by the spatial attention network form four levels of feature maps, thereby forming a multi-scale feature pyramid (i.e., multi-scale attention features), which provides more spatial context information for the decoding of the subsequent decoder to decode the multi-scale attention features to obtain the binocular camera pose transformation information and image target change information between adjacent frame images.

[0032] The embodiment extracts multi-level features of adjacent frame images through the hop-connected codec module to provide more space-time information.

[0033] In one embodiment, the codec module includes a pose transformation codec, an optical flow codec, and an appearance flow codec; the binocular camera pose transformation information is a binocular pose transformation matrix; the image target change information includes an optical flow map and an appearance flow map; wherein: The decoder of the pose transformation codec is configured to perform dimension compression processing and feature refinement processing on the features output by the encoder of the pose transformation codec to obtain the binocular pose transformation matrix. The decoder of the optical flow codec is configured to decode the multi-scale attention features output by the spatial attention network of the optical flow codec to obtain the optical flow map. The appearance flow codec is configured to extract features from the left adjacent frame image and the optical flow map to obtain the appearance flow map.

[0034] Specifically, the codec module in the spatiotemporal information extraction network includes at least a codec for generating stereo camera pose transformation information between adjacent frames (i.e., a pose transformation codec, including a pose transformation encoder and a pose transformation decoder), a codec for generating optical flow graphs between adjacent frames (i.e., an optical flow codec, including an optical flow encoder and an optical flow decoder), and a codec for generating appearance flow graphs between adjacent frames (i.e., an appearance flow codec, including an appearance flow encoder and an appearance flow decoder). The stereo camera pose transformation information can be a stereo pose transformation matrix; the image target change information includes optical flow graphs and appearance flow graphs.

[0035] The features output by the pose transformation encoder are further processed by a spatial attention network to extract multi-scale features. The resulting multi-scale attention features are then input into the pose transformation decoder. The pose decoder first compresses the dimensionality of the multi-scale attention features, and then refines the features step by step through a three-level convolutional network to obtain multi-degree-of-freedom pose parameters. The stereo pose transformation matrix is ​​then obtained from the multi-degree-of-freedom pose parameters.

[0036] The encoding and decoding process of the optical flow encoder and decoder is similar to that of the pose transformation encoder described above. The optical flow decoder decodes the multi-scale attention features (generated by the spatial attention network in the optical flow encoder and decoder module) to obtain the optical flow map.

[0037] Depend on Figure 2 It can be seen that the optical flow graph output by the optical flow encoder-decoder module and the left adjacent frame image (from the input spatiotemporal information extraction network) are jointly input into the appearance flow encoder so that the appearance flow decoder outputs the appearance flow graph.

[0038] This embodiment uses a skip-connected codec module to fuse multi-scale coding features of adjacent frame images, thereby obtaining more spatiotemporal information from adjacent frame images.

[0039] In one embodiment, the spatial attention network includes a texture feature extraction network, an edge awareness enhancement network, a reflective region suppression network, a channel stitching module, and a residual connection network, wherein: The texture feature extraction network is used to extract texture features from the initial encoded features to obtain image texture features; The edge-aware enhancement network is used to extract edge features from the initial encoded features to obtain image edge features; The reflective region suppression network is used to extract reflective suppression features from the initial encoded features to obtain image reflective suppression features; The channel stitching module is used to perform channel-dimensional stitching processing on the image texture features, the image edge features, and the image reflection suppression features to obtain an attention map; The residual connection network is configured to perform residual connection on the attention map and the initial encoding feature to obtain a multi-scale attention feature.

[0040] Specifically, as shown in FIG. 1, the spatial attention network added after the ResNet18 includes a texture feature extraction network, an edge perception enhancement network, a specular region suppression network, a channel splicing module, and a residual connection network. Figure 3 conv1x1 is a normal 1x1 convolution (using a 1x1 size convolution kernel / filter to perform full-channel linear weighted summation on each pixel / spatial position of the input feature map); conv3x3 is a depth separable convolution (including a depth convolution for processing spatial information and a point-by-point convolution for processing channel information, the depth separable convolution decouples spatial and channel information through step-by-step operation of space first and then channel, thereby reducing the amount of calculation; a rectified linear unit (ReLU) is an activation function; and a sigmoid is another activation function.

[0041] The process of obtaining the multi-scale attention feature by the spatial attention network is shown in formula 1, wherein, is the multi-scale attention feature; is a depth separable convolution with a convolution kernel size of is Figure 3 is the original feature input into the spatial attention network, i.e., the initial encoding feature; is the attention map; is element-wise multiplication; is element-wise addition.

[0042] ; (1) ; (2) wherein, The generated fusion combines multi-source feature information (image texture feature, image edge feature, and image specular suppression feature), as shown in formula 2, wherein, is a sigmoid activation function; is an image texture feature; is an image edge feature; is an image specular suppression feature; is a channel dimension splicing operation. Wherein, Figure 2 The Sobel Kernel in the formula is a tool for edge detection in computer vision, which calculates the image gradient (gray change rate) through convolution operation, and quickly locates the area of gray level mutation in the image, i.e., the edge.

[0043] ​This embodiment uses multi-scale texture feature extraction, edge awareness enhancement, and reflective area suppression to accurately allocate attention to key regions in adjacent frame images.

[0044] In one embodiment, the step of extracting features from the adjacent frame images of the left eye and the optical flow map to obtain the appearance flow map includes: Obtain the brightness change value of each pixel in the adjacent frame image of the left eye; Based on the brightness change value and the geometric deformation field, the luminance change value of each pixel is determined; the geometric deformation field is determined based on the optical flow map. The appearance flow map is determined based on the luminance variation values ​​of each pixel.

[0045] Specifically, the process of the appearance stream codec generating the appearance stream graph is shown in Equation 3, where, for Pixels in adjacent frames at different times The brightness value; For geometric deformation fields, describing the pixel from arrive The displacement is determined by the optical flow map. The resulting appearance flow map specifically includes pixels. The change in luminance from the previous frame to the current frame is caused by changes in illumination and non-Lambertian reflection (the phenomenon of light reflection from the surface of a real object, where the intensity of reflected light changes with the viewing direction / angle of view or the angle of incidence). .

[0046] (3) like Figure 2 As shown, the inputs to the pose transformation codec and the optical flow codec are... and ,as well as and Two sets of adjacent frames for the left eye, among which, and The time frame number of the image; The left eye is represented by the optical flow map output by the optical flow codec. The pixel displacement is represented by a two-dimensional vector and superimposed on the coordinates of adjacent frames (pixels) to form a sampling grid. Then, bilinear interpolation resampling is used to synthesize a registration image (the image of the adjacent frame of the left eye and the optical flow map). The registration image is input into the appearance flow codec to generate the appearance flow map.

[0047] This embodiment generates an appearance flow graph by analyzing the changes in pixels in adjacent frame images.

[0048] In one embodiment, the binocular stereo matching network includes a multi-layer feature extraction module, a stereo matching module, and a self-attention module, wherein: The multi-layer feature extraction module is configured to extract multi-layer features of the corrected binocular images. The stereo matching module is configured to determine a feature similarity matrix based on the multi-layer features. The self-attention module is configured to perform regional propagation processing on a multi-level disparity map to obtain a multi-stage disparity map, wherein the multi-level disparity map is generated based on the feature similarity matrix.

[0049] Specifically, the left and right stereo corrected images are adjusted to a specified size as the input of the binocular stereo matching network. First, a convolutional neural network (CNN) is used to extract global features, and the extracted global features are down-sampled by multiple times, and position encoding is added to the compressed pixel features after down-sampling.

[0050] Then, the features with added position encoding are input into a multi-level Transformer (feature extraction unit, i.e., multi-layer feature extraction module) to extract multi-layer features. A parallel softmax stereo matching layer (i.e., stereo matching module) is directly connected after each Transformer to calculate a feature similarity matrix to establish a pixel-level correspondence relationship to avoid the limitation of the preset disparity range. Based on the feature similarity matrix between the corrected binocular images, three disparity maps with different levels of characteristics are generated.

[0051] The three disparity maps are input into a weight-shared self-attention module to jointly optimize the multi-level predictions, and the more accurate prediction results of the reliable area (area with little change) are propagated to the occluded area (area with large change) by using the structural similarity of the corrected binocular images themselves.

[0052] The embodiment analyzes the structural similarity of the binocular images at the same time by using the binocular stereo matching network to obtain a multi-stage disparity map for generating a depth map.

[0053] Specifically, the depth estimation model provided by the present application further includes a loss function part, as shown in Figure 2 The loss function specifically includes a re-projection perception loss , a photometric calibration loss , a stereo matching loss , and a disparity smoothing loss .

[0054] The re-projection perception loss serves to improve the accuracy of depth estimation and camera pose estimation through multi-view geometric consistency constraints. The method of warping the image frames at the previous and next time to the current image frame is shown in formula 4, wherein, is the current image frame; is the image frame at the previous and next time; is the depth map output by the depth evaluation network; For to the relative camera pose between the left and right images; is a camera intrinsic matrix; is a bilinear sampling operation.

[0055] ; (4) The accuracy of depth estimation and camera pose estimation is improved by re-projection perception loss (adjusting the parameters of the depth estimation model). A feature space-based perception loss is used, which does not directly calculate the pixel intensity difference between adjacent image frames, but inputs the warped image frame and the corresponding into the Transformer to obtain a feature map.

[0056] The photometric calibration loss reduces the influence of photometric inconsistency. The stereo matching loss is calculated by calculating the pixel intensity loss and the structural similarity (SSIM) loss between the left image synthesized by the right image and the corresponding real left image Disparity smoothing loss is used to maintain the reasonableness of the disparity map structure and solve the ill-conditioned area of the image.

[0057] ; (5) The total loss function obtained by the above-mentioned multiple loss functions is shown in formula 5. Wherein, , , and are the weights (hyperparameters) of the corresponding loss functions.

[0058] The image depth estimation device provided by the present application is described below. The image depth estimation device described below can be mutually corresponding to the image depth estimation method described above.

[0059] Please refer to Figure 4 , the present application also provides an image depth estimation device, comprising: An image depth evaluation module 401 is configured to input left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images include a left eye adjacent frame image and a corrected binocular image; Wherein, the depth estimation model comprises a space-time information extraction network, a binocular stereo matching network and a depth evaluation network; The space-time information extraction network comprises a plurality of skip-connected codec modules, and each codec module is configured to: extracting spatio-temporal features from the left-eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; The binocular stereo matching network is configured to perform multi-layer feature extraction on the corrected binocular image to obtain multi-level disparity maps. The depth evaluation network is configured to evaluate the binocular camera pose transformation information, the image target change information, and the multi-level disparity maps to obtain a depth map of the left-eye and right-eye images.

[0060] Optionally, the codec module comprises an encoder, a spatial attention network, and a decoder. The encoder is configured to obtain initial encoded features of the left-eye adjacent frame image. The spatial attention network is configured to perform texture feature extraction, edge perception enhancement, and anti-reflection region suppression on the initial encoded features to obtain multi-scale attention features. The decoder is configured to decode the multi-scale attention features to obtain binocular camera pose transformation information and image target change information between adjacent frame images.

[0061] Optionally, the codec module comprises a pose transformation codec, an optical flow codec, and an appearance flow codec; the binocular camera pose transformation information is a binocular pose transformation matrix; the image target change information comprises an optical flow map and an appearance flow map; and wherein: The decoder of the pose transformation codec is configured to perform dimension compression processing and feature refining processing on the features output by the encoder of the pose transformation codec to obtain the binocular pose transformation matrix. The decoder of the optical flow codec is configured to decode the multi-scale attention features output by the spatial attention network of the optical flow codec to obtain the optical flow map. The appearance flow codec is configured to perform feature extraction on the left-eye adjacent frame image and the optical flow map to obtain the appearance flow map.

[0062] Optionally, the spatial attention network comprises a texture feature extraction network, an edge perception enhancement network, an anti-reflection region suppression network, a channel concatenation module, and a residual connection network, wherein: The texture feature extraction network is configured to perform texture feature extraction on the initial encoded features to obtain image texture features. The edge perception enhancement network is configured to perform edge feature extraction on the initial encoded features to obtain image edge features. The anti-reflection region suppression network is configured to perform anti-reflection suppression feature extraction on the initial encoded features to obtain image anti-reflection suppression features. The channel splicing module is configured to splice the image texture feature, the image edge feature and the image anti-reflection feature in a channel dimension to obtain an attention map. The residual connection network is configured to perform residual connection on the attention map and the initial encoding feature to obtain a multi-scale attention feature.

[0063] Optionally, the appearance flow codec is further configured to: obtain a luminance change value of each pixel in the left-eye adjacent frame image; determine a radiance change value of each pixel based on the luminance change value and a geometric deformation field, the geometric deformation field being determined based on the optical flow map; determine the appearance flow map based on the radiance change value of each pixel.

[0064] Optionally, the binocular stereo matching network comprises a multi-layer feature extraction module, a stereo matching module and a self-attention module, wherein: The multi-layer feature extraction module is configured to extract multi-layer features of the corrected binocular image. The stereo matching module is configured to determine a feature similarity matrix based on the multi-layer features. The self-attention module is configured to perform regional propagation processing on a multi-level disparity map to obtain a multi-stage disparity map, the multi-level disparity map being generated based on the feature similarity matrix.

[0065] Figure 5 An example of a schematic diagram of an entity structure of an electronic device is shown in FIG. 1. Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute an image depth estimation method, which includes inputting left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images include a left eye adjacent frame image and a corrected binocular image; wherein the depth estimation model includes a space-time information extraction network, a binocular stereo matching network, and a depth evaluation network; the space-time information extraction network includes a plurality of skip-connected codec modules, each codec module being configured to: perform space-time feature extraction on the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; the binocular stereo matching network is configured to perform multi-layer feature extraction on the corrected binocular image to obtain a multi-level disparity map; and the depth evaluation network is configured to evaluate the binocular camera pose transformation information, the image target change information, and the multi-level disparity map to obtain a depth map of the left and right eye images.

[0066] In addition, the logical instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0067] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer readable storage medium, and the computer program being executable by a processor to cause a computer to perform the image depth estimation method provided by any of the above methods, the method comprising: inputting left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images comprising a left eye adjacent frame image and a corrected binocular image; wherein the depth estimation model comprises a space-time information extraction network, a binocular stereo matching network and a depth evaluation network; the space-time information extraction network comprises a plurality of skip-connection codec modules, each codec module being configured to: perform space-time feature extraction on the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; the binocular stereo matching network is configured to perform multi-layer feature extraction on the corrected binocular image to obtain a multi-level disparity map; and the depth evaluation network is configured to evaluate the binocular camera pose transformation information, the image target change information and the multi-level disparity map to obtain the depth map of the left and right eye images.

[0068] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the image depth estimation method provided by any of the above methods, the method comprising: inputting left and right eye images into a depth estimation model to obtain a depth map output by the depth estimation model; the left and right eye images comprising a left eye adjacent frame image and a corrected binocular image; wherein the depth estimation model comprises a space-time information extraction network, a binocular stereo matching network and a depth evaluation network; the space-time information extraction network comprises a plurality of skip-connection codec modules, each codec module being configured to: perform space-time feature extraction on the left eye adjacent frame image to obtain binocular camera pose transformation information and image target change information between adjacent frame images; the binocular stereo matching network is configured to perform multi-layer feature extraction on the corrected binocular image to obtain a multi-level disparity map; and the depth evaluation network is configured to evaluate the binocular camera pose transformation information, the image target change information and the multi-level disparity map to obtain the depth map of the left and right eye images.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0070] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0071] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image depth estimation method, characterized in that, include: The left and right eye images are input into the depth estimation model to obtain the depth map output by the depth estimation model; the left and right eye images include the left eye's adjacent frame image and the corrected binocular image; The depth estimation model includes a spatiotemporal information extraction network, a binocular stereo matching network, and a depth evaluation network. The spatiotemporal information extraction network includes multiple skip-connected encoder-decoder modules, each of which is used for: Spatiotemporal features are extracted from the adjacent frames of the left eye to obtain the binocular camera pose transformation information and image target change information between adjacent frames; The binocular stereo matching network is used to perform multi-level feature extraction on the corrected binocular image to obtain a multi-level disparity map; The depth evaluation network is used to evaluate the pose transformation information of the binocular camera, the image target change information, and the multi-level disparity map to obtain the depth maps of the left and right eye images.

2. The image depth estimation method according to claim 1, characterized in that, The encoder-decoder module includes an encoder, a spatial attention network, and a decoder; The encoder is used to obtain the initial encoding features of the left-eye adjacent frame image; The spatial attention network is used to extract texture features, enhance edge perception, and suppress reflective areas from the initial encoded features to obtain multi-scale attention features. The decoder is used to decode the multi-scale attention features to obtain the binocular camera pose transformation information and image target change information between adjacent frame images.

3. The image depth estimation method according to claim 2, characterized in that, The codec module includes a pose transformation codec, an optical flow codec, and an appearance flow codec; the binocular camera pose transformation information is a binocular pose transformation matrix; the image target change information includes an optical flow map and an appearance flow map; wherein: The decoder of the pose transformation codec is used to perform dimensionality compression and feature refinement on the features output by the encoder of the pose transformation codec to obtain the binocular pose transformation matrix. The decoder of the optical flow codec is used to decode the multi-scale attention features output by the spatial attention network of the optical flow codec to obtain the optical flow map; The appearance stream codec is used to extract features from the adjacent frame images of the left eye and the optical flow map to obtain the appearance flow map.

4. The image depth estimation method according to claim 2, characterized in that, The spatial attention network includes a texture feature extraction network, an edge awareness enhancement network, a reflective region suppression network, a channel stitching module, and a residual connection network, wherein: The texture feature extraction network is used to extract texture features from the initial encoded features to obtain image texture features; The edge-aware enhancement network is used to extract edge features from the initial encoded features to obtain image edge features; The reflective region suppression network is used to extract reflective suppression features from the initial encoded features to obtain image reflective suppression features; The channel stitching module is used to perform channel-dimensional stitching processing on the image texture features, the image edge features, and the image reflection suppression features to obtain an attention map; The residual connection network is used to perform residual connections between the attention map and the initial encoded features to obtain multi-scale attention features.

5. The image depth estimation method according to claim 3, characterized in that, The step of extracting features from the adjacent frame images of the left eye and the optical flow map to obtain the appearance flow map includes: Obtain the brightness change value of each pixel in the adjacent frame image of the left eye; Based on the brightness change value and the geometric deformation field, the luminance change value of each pixel is determined; the geometric deformation field is determined based on the optical flow map. The appearance flow map is determined based on the luminance variation values ​​of each pixel.

6. The image depth estimation method according to claim 1, characterized in that, The binocular stereo matching network includes a multi-layer feature extraction module, a stereo matching module, and a self-attention module, wherein: The multi-layer feature extraction module is used to extract multi-layer features from the corrected binocular image; The stereo matching module is used to determine the feature similarity matrix based on the multi-layer features; The self-attention module is used to perform region propagation processing on the multi-level disparity map to obtain a multi-level disparity map; the multi-level disparity map is generated based on the feature similarity matrix.

7. An image depth estimation device, characterized in that, include: The image depth evaluation module is used to input the left and right eye images into the depth estimation model to obtain the depth map output by the depth estimation model; the left and right eye images include the left eye adjacent frame image and the corrected binocular image; The depth estimation model includes a spatiotemporal information extraction network, a binocular stereo matching network, and a depth evaluation network. The spatiotemporal information extraction network includes multiple skip-connected encoder-decoder modules, each of which is used for: Spatiotemporal features are extracted from the adjacent frames of the left eye to obtain the binocular camera pose transformation information and image target change information between adjacent frames; The binocular stereo matching network is used to perform multi-level feature extraction on the corrected binocular image to obtain a multi-level disparity map; The depth evaluation network is used to evaluate the pose transformation information of the binocular camera, the image target change information, and the multi-level disparity map to obtain the depth maps of the left and right eye images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image depth estimation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image depth estimation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image depth estimation method as described in any one of claims 1 to 6.

Images