Method and system for dense regression tasks

Abstract

To provide a method.SOLUTION: A computer-implemented method for dense regression tasks in input data (I) comprises: (S10) determining at least one encoded feature map (F) of the input data (I); (S20) discretizing the at least one encoded feature map (F) to obtain discrete representations (H) configured to represent internal space patterns of the input data (I); (S30) extracting pixel-level features (P) for each element of the at least one encoded feature map (F); and (S40) calculating a predicted output (D) by transferring a part of the discrete representations (H) to locations of the pixel-level features (P). The part and the locations are calculated based on similarity between elements of the pixel-level features (P) and elements of the discrete representations (H).SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] 1. Field of the Invention The present invention relates to a method and system for dense regression tasks, in particular to a method and system for depth estimation in images.

[0002] 2. Description of Related Technology The task of dense regression, i.e., providing estimates of the values ​​of parameters for each element of input data, is important in many applications.

[0003] In particular, depth estimation is a dense regression task that is of paramount importance in computer vision, for example, to understand geometric relationships in a scene. Such a dense regression task consists of predicting the distance between the center of projection of an image and the geometric location corresponding to each pixel. Depth estimation finds direct importance in downstream applications such as 3D modeling, robotics, and autonomous vehicles. The estimation is generally considered dense because it is achieved for each pixel of the input image.

[0004] Among depth estimation tasks, the monocular depth estimation task, i.e., estimating the depth corresponding to pixels from a single image, is particularly challenging because the inherent scale ambiguity means that the same 2D input image with a given set of pixels can correspond to an infinite number of 3D scenes with different sets of geometric locations.

[0005] Recent advances in the use of neural networks have led to significant improvements in the performance of monocular depth estimation tasks. In particular, state-of-the-art methods typically involve convolutional neural networks (CNNs).

[0006] However, most methods use neural network models that impose geometric constraints on the image or explicitly discretize the continuous output depth using a predefined range. Such geometric constraints and predefined depth ranges can be considered as imposing prior knowledge, which prevents the model from modeling arbitrary patterns with arbitrary depths that are common in real-world scenes. Thus, such prior knowledge inherently limits the expressive power of the method.

[0007] Therefore, there is a need for a method for dense regression tasks that overcomes the limitations of the prior art.

[0008] The following references disclose various methods for the task of monocular depth estimation using neural networks or techniques useful for performing such tasks. [Reference 1] BHAT, Shariq Farooq, ALHASHIM, Ibraheem, and WONKA, Peter. Adabins: Depth estimation using adaptive bins. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4009-4018. [Reference 2] FU, Huan, GONG, Mingming, WANG, Chaohui et al. Deep ordinal regression network for monocular depth estimation. Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, p. 2002-2011. [Reference 3] LEE, Jin Han, HAN, Myung-Kyu, KO, Dong Wook et al. From big to small: Multi-scale local planar guidance for monocular depth estimation.arXiv preprint arXiv:1907.10326, 2019. [Reference 4] MASOUMIAN, Armin, RASHWAN, Hatem A., CRISTIANO, Julian et al., Monocular depth estimation using deep learning: A review, Sensors, 2022, vol. 22, no. 14, p. 5353. [Reference 5] ZHU, Xizhou, SU, Weijie, LU, Lewei, LI Bin, WANG, Xiaogang, and DAI, Jifeng, Deformable DETR: deformable transformers for end-to-end object detection.9th International Conference on Learning Representations, ICLR2021, Virtual Event, Austria, May 3-7, 2021. [Reference 6] BA, Jimmy Lei, KIROS, Jamie Ryan, and HINTON, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [Reference 7] Geiger, Andreas, Lenz, Philip, and Urtasun, Raquel. Are we ready for autonomous driving? The KITTI vision benchmark suite. 2012 IEEE conference on computer vision and pattern recognition, IEEE, 2012, pp. 3354-3361. [Reference 8] ZHANG, Zhenyu, CUI, Zhen, XU, Chunyan et al., Pattern-affinitive propagation across depth, surface normal and semantic segmentation. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, p.4106-4115. [Reference 9] PATIL, Vaishakh, SAKARIDIS, Christos, LINIGER, Alexander et al. P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, p. 1610 - 1621. [Reference 10] YIN, Wei, LIU, Yifan, SHEN, Chunhua et al. Enforcing geometric constraints of virtual normal for depth prediction. Proceedings of the IEEE / CVF International Conference on Computer Vision and Pattern Recognition, 2019, p. 5684 - 5693. [Reference 11] YU, Zehao, ZHENG, Jia, LIAN, Dongze et al. Single - image piece - wise planar 3d reconstruction via associative embedding. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, p. 1029 - 1037. [Reference 12] LEE, Sihaeng, LEE, Janghyeon, KIM, Byungju et al. Patch - wise attention network for monocular depth estimation. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, p. 1873 - 1881. [Reference 13] QIAO, Siyuan, ZHU, Yukun, ADAM, Hartwig et al., Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, p.3997-4008. [Reference 14] YUAN, Weihao, GU, Xiaodong, DAI, Zuozhuo et al., Neural Window Fully-Connected CRFs for Monocular Depth Estimation. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, p.3916-3925. [Reference 15] AGARWAL, Ashutosh and ARORA, Chetan, Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention. Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2023, pp. 5861-5870. [Reference 16] EIGEN, David, PUHRSCH, Christian, and FERGUS, Rob. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 2014, vol. 27. Summary of the Invention

[0009] According to an aspect of the present disclosure, there is provided a computer-implemented method for a dense regression task on input data, the method comprising: - determining at least one encoding feature map of the input data; - discretizing at least one encoding feature map to obtain a discrete representation configured to represent an internal spatial pattern of the input data; - extracting pixel-level features for each element of at least one encoding feature map; - calculating a predicted output by transferring a portion of the discrete representation to a location of pixel-level features, the portion and location being calculated based on the similarity between elements of the pixel-level features and elements of the discrete representation.

[0010] For the sake of brevity, a computer-implemented method for the dense regression task in images will be referred to hereinafter as a regression method, and a system for the dense regression task in images will be referred to hereinafter as an estimation system.

[0011] As used herein, a dense regression task may comprise, inter alia, any computer vision task involving the estimation of one or more values ​​for each pixel of an image, which may comprise, for example, depth values, surface normal data, optical flow, and boundary detection.

[0012] Herein, an encoding feature map of input data is a mathematical object that comprises the (high-order) features of the input data encoded in higher dimensions. Furthermore, the encoding feature map generally has a lower spatial extent than the input data.

[0013] The encoding feature map can be obtained by processing input data through one or more layers of a neural network. For example, the encoding feature map can be obtained through a convolution layer of a convolutional neural network. In particular, the encoding feature map can be obtained through a series of nonlinearities, trainable convolution layers, and normalization layers of a convolutional neural network.

[0014] As used herein, the location of a pixel-level feature is the spatial region within the array that represents the pixel-level feature.

[0015] Herein, each discrete representation represents a particular kind of internal spatial pattern in the input data, also referred to as an internal spatial concept. In an image, the internal spatial pattern can be, for example, a general high-level feature such as objectness (the likelihood that an object is present in the proposed region of interest), relative positioning, geometric structure, etc.

[0016] In contrast to prior art methods, the regression method of the present disclosure generates a discrete representation of an image at an internal level. The discretization occurs at the abstract level of the image's internal features, rather than at the output level (e.g., the output depth level). Therefore, such a discrete representation is less constrained than representations obtained using other methods, whose output is constrained by the limited variety of objects that can be used as prior knowledge (starting points) for the output.

[0017] Because the method removes explicit geometric constraints on the scene represented by the input image, such a method outperforms most existing methods on known datasets and benchmarks.

[0018] According to an aspect, the steps of the method are performed for a plurality of encoding feature maps, each of the plurality of encoding feature maps having a resolution parameter corresponding to a resolution of the input data to obtain a plurality of predicted outputs, and the method further includes combining the plurality of predicted outputs obtained from the plurality of encoding feature maps having different resolutions.

[0019] This configuration allows for the collection of information from different resolutions, which improves the efficiency of the method. In particular, when the input data is an image, the encoding feature maps at lower resolutions may capture the relative positions of objects and surfaces, e.g., background and foreground, while the encoding feature maps at higher resolutions capture more precise details such as object boundaries.

[0020] According to an aspect, combining a plurality of predicted outputs obtained from a plurality of encoded feature maps having different resolutions includes calculating an average value of the plurality of predicted outputs.

[0021] According to an aspect, extracting pixel-level features for each element of the at least one encoding feature map includes calculating a plurality of pixel-level features having different resolutions.

[0022] According to an aspect, determining at least one encoding feature map of the input data includes using a multi-scale deformable attention module to refine the encoding feature maps at different resolutions.

[0023] Thus, information between different resolutions is shared particularly efficiently.

[0024] According to an aspect, discretizing the at least one encoded feature map to obtain a discrete representation configured to represent an internal spatial pattern of the input data includes using a cross-attention module.

[0025] According to an aspect, the cross-attention module is a transposed cross-attention module configured to encourage soft clustering of coding features among the discrete representations.

[0026] Therefore, the attention weights of the cross-attention module, which are input-dependent in nature, can be used to partition the encoded feature maps into separate discrete representations in a particularly efficient way.

[0027] Therefore, in contrast to many prior art methods, the discretization process of the regression method of the present disclosure can be input dependent and does not need to be predetermined during the design of the method. Furthermore, specific discretization of feature maps between different discrete representations occurs with similar inference times.

[0028] According to an aspect, the transposed cross-attention module comprises learnable prior knowledge, and the discrete representations are iteratively obtained using the transposed cross-attention module initialized with the learnable prior knowledge.

[0029] Thus, the prior knowledge of the discretization process can be trained and improved with each iteration of the process, providing better performance for estimation tasks compared to methods using cross-attention modules with non-learnable prior knowledge, such as non-learnable attention modules with random initialization, where the hidden representations do not embody task-specific prior information.

[0030] According to an aspect, the step of computing the (non-projected) predicted output by transferring a portion of the discrete representation to pixel-level feature locations includes using a cross-attention module.

[0031] According to an aspect, the method further includes projecting the predicted output to a physical value.

[0032] According to an aspect, the dense regression task is a monocular depth estimation task in an image, where the predicted output is projected onto depth values ​​to obtain a depth map.

[0033] The inventors have observed that the method is particularly efficient when applied to monocular depth estimation in images. In particular, by discretizing the image at an internal level, the method more efficiently captures the structure of the scene and therefore estimates depth. This is particularly noticeable in indoor scenes, since indoor scenes typically have a large number of objects present.

[0034] We have shown that our method outperforms most known methods when used on typical benchmark datasets, such as the KITTI dataset disclosed in [Reference 6]. Furthermore, we have observed that our method is particularly efficient when applied to dense regression tasks such as surface normal estimation.

[0035] According to an aspect, a system for dense regression tasks on input data is disclosed, the system comprising one or more processors and a computer-readable medium, the computer-readable medium comprising computer instructions stored therein, the instructions, when executed by the one or more processors, causing the one or more processors to: - determining, using an encoder module, at least one encoding feature map of the input data; - discretizing, using a first discretization module, at least one encoding feature map to obtain a discrete representation configured to represent an internal spatial pattern of the input data; using a decoder module to extract pixel-level features from at least one encoded feature map; - using a second discretization module to transfer a portion of the discrete representation to a location of a pixel-level feature, the portion and location being calculated based on the similarity between elements of the pixel-level feature and elements of the discrete representation, thereby calculating a predicted output.

[0036] The present disclosure further relates to a computer program comprising instructions for carrying out the steps of the regression method when said program is executed by a computer, said program being in any programming language and may be in the form of source code, object code, or a code intermediate between source code and object code, for example in a partially compiled form, or in any other desired form.

[0037] The present disclosure further relates to a computer-readable recording medium having recorded thereon a computer program comprising instructions for performing the steps of the above-mentioned method. The recording medium may be any entity or device capable of storing a program. For example, the medium may comprise a storage means such as a ROM, e.g., a CD ROM or a microelectronic circuit ROM, or a magnetic storage means, e.g., a diskette (floppy disk) or a hard disk.

[0038] Alternatively, the recording medium may be an integrated circuit having the program embedded therein, the circuit being configured to perform, or be used in the performance of, the method. [Brief explanation of the drawings]

[0039] The features, advantages, and technical and industrial importance of preferred embodiments of the present invention will be described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Figure 1] FIG. 1 is a diagram illustrating aspects of a method for dense regression tasks according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a schematic representation of an architecture for a method for dense regression tasks according to an embodiment of the present disclosure. [Figure 3A] FIG. 3A is a chart illustrating the advantages of an embodiment of a method for dense regression tasks according to the present disclosure over other methods in the prior art. [Figure 3B] FIG. 3B is a chart illustrating the advantages of an embodiment of a method for dense regression tasks according to the present disclosure over other methods in the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0040] Embodiments of the regression method will be described with reference to Figures 1 and 2, respectively. While the described embodiments relate in particular to methods for monocular depth estimation tasks in images, the present disclosure is not limited to such embodiments, but also relates to other dense regression tasks, such as, for example, surface normal estimation tasks. Furthermore, the present disclosure also relates to dense regression tasks where the input data is not an image, but any data, such as data with elements in a regular grid.

[0041] Referring to FIG. 1 , an embodiment of the dense regression method may include the following major processing steps: determining an encoding feature map F of image I (S10); discretizing the encoding feature map F to obtain a discrete representation H configured to represent the internal spatial pattern of image I (S20); extracting pixel-level features P for each pixel in the image (S30); transferring a portion of the discrete representation H to the pixel-level features P (S40); and projecting the predicted output to physical values ​​(S50).

[0042] The method may take as input data an image, for example an RGB image having two spatial dimensions (x, y) and three color dimensions (R, G, B), and may provide as output a depth map, i.e., an image with pixels each associated with a depth value corresponding to an estimated depth of the geometric location imaged by the corresponding pixel in the input image.

[0043] S10: Determine the coding feature map (F) of the image (I). In step S10, an encoding feature map F of the input image I is calculated. This calculation can be performed by different means. For example, the encoding feature map F can be calculated using an encoder module. The encoder module can be, for example, convolutional or attention-based. In particular, the encoder used can have a structure that increases the receptive field of either convolutional or windowed attention by reducing the resolution of the encoding feature map.

[0044] In an embodiment, the encoder module used may be based on a classification module, with its last three layers (pooling layer, fully connected layer, and softmax layer) removed.

[0045] According to an embodiment, the encoder module generates encoded feature maps F for different resolutions l, i.e., corresponding to different resolutions or scales, for an image l. l The resolution of the encoded feature map is generally lower than the size of the input image, and may be, for example, equal to 1 / 32 of the size of the input image.

[0046] In embodiments, the encoding features at different resolutions generated by the encoder module may be refined, and information between the encoding features at different resolutions may be shared, for example, via one or more multi-scale deformable attention modules (MSDA), such as those disclosed in [Reference 5]. In particular, the encoding feature maps at different resolutions may be combined via a feature pyramid network utilizing layer normalization, as described in [Reference 6].

[0047] S20: Discretize the encoded feature map (F) to obtain a discrete representation (H). In step S20, the coding feature map F of the image I obtained from step S10 is discretized to obtain a discrete representation of the image. The discretization can be implemented using a discretization module, for example, an arbitrary feature partition (AFP) module. The number of possible discrete representations can have different values. For example, the inventors have observed that it is advantageous to select a number of discrete representations comprised between 8 and 32. Furthermore, the inventors have observed that it is appropriate to avoid a large number of discrete representations. In particular, the inventors have observed that a number of discrete representations close to the number of coding features in the coding feature map will generate noise in the discretization process, thereby reducing its efficiency.

[0048] According to an embodiment, each encoded feature map at a different resolution (e.g., obtained from an MSDA encoder) is discretized to obtain multiple discrete representations at the same different resolutions. In such a case, the discretization may also be performed using an AFP module.

[0049] The discretization of the encoding feature maps at different resolutions can be handled independently so that it is sufficient to describe the discretization implemented for a single resolution, and it will be clear to one skilled in the art how to apply the same process for encoding feature maps at other resolutions.

[0050] The AFP module may be a neural network such as a cross-attention module that implements discretization by learning how to define the partitioning functions in an input-dependent manner, i.e., depending on the encoding feature maps input in the AFP module.

[0051] Specifically, the cross-attention module includes attention weights that are used to derive partitions of the encoding feature map between different discrete representations. In particular, for an encoding feature map, the attention weights represent the probability of belonging to a certain partition (discrete representation).

[0052] Because attention weights, by definition, inherently depend on the AFP module's input, i.e., the encoding feature maps, the specific partitioning also depends on the input and occurs at similar inference time. As a result, the AFP module provides soft, mutually exclusive partitioning for the encoding feature maps. Thus, each encoding feature map is effectively assigned to a single, discrete representation. Furthermore, such exclusivity allows for the use of a smaller number of partitions for the encoding feature maps.

[0053] According to an embodiment, the cross-attention module is equipped with learnable prior knowledge and is referred to as a learnable prior cross-attention module. Given a given encoded feature map, such a learnable prior cross-attention module performs an iterative process of refining an initial discrete representation used as prior knowledge, over an iteration number R. Thus, in our method, the prior knowledge is an input-dependent partition initialization. This is in stark contrast to prior art methods, where the module's prior knowledge (such as geometric prior knowledge) is output prior knowledge handcrafted at the image level.

[0054] Specifically, in an embodiment, the learnable cross-attention module is a discrete representation of the following softmax(KQ T )] T V can be a transposed cross-attention module obtained using a transposed softmax operation such as (Equation 1), where K and V are the query tensor and key tensor obtained from the encoded feature map, and Q is the sum of the previous discrete representation H prior is a value tensor initialized to . The transpose operation provides a way to achieve soft mutual exclusivity between discrete representations.

[0055] The term "transpose" refers to the following softmax(KQ T ) (Equation 2).

[0056] Such tensors can be obtained as projections of feature maps and prior internal discrete representations.

[0057] Following the discretization process, the encoding feature map is partitioned between discrete representations using a learnable prior knowledge transposition cross-attention module. At the t-th iteration (or in other words, refinement step t) of R iterations, the (query) discrete representation sought for a given value of the encoding feature map can be expressed as follows:

number

number

number

number

number

number

[0058] According to an embodiment, attention weights may be normalized to 1 along the "i" dimension to avoid disappearing or exploding quantities due to summation of non-normalized distributions.

[0059] Advantageously, the inherent behavior of the transposed softmax function forces competition between outputs, which encourages quantization partitioning between different discrete representations. Specifically, due to competition, weights may be large only relative to other losses. Thus, for a given coding feature, there may be few attention weights significantly greater than zero. This forces the value v, i.e., the output-related content of the coding feature, to be routed to only some discrete representations in successive iterations. The encouraged competition forces soft clustering of the input coding feature map, where the actual partitioning function is represented by the last iteration query. In other words, the discrete representation may be determined by the query obtained after the last iteration of the soft attention module.

[0060] S30 Extract pixel-level features (P) In step S30, pixel-level features (P) are extracted from the encoded feature map (F). The extraction may be implemented using the decoder module (120), for example, a feature pyramid network (FPN) module.

[0061] According to an embodiment, the input data is an image and the decoder module maps the low-resolution encoded feature maps to the full input resolution feature maps in a pixel-by-pixel regression.

[0062] S40 Calculate the predicted output by converting part of the discrete representation (H) into pixel-level features (P) In step S40, it is assumed that a portion of the discrete representation is transferred to pixel-level features, thereby obtaining a predicted output. Step S40 may be implemented, for example, using an internal scene discretization module (150). The internal scene discretization module (150) receives as input the discrete representation obtained from step S20 and the pixel-level features obtained from step S30.

[0063] According to an embodiment, both the portion of the discrete representation that is transferred to the pixel-level features and the location where the discrete representation is transferred are calculated based on the similarity between the discrete representation and the pixel-level features.

[0064] Specifically, for images, the similarity between the discrete representation and pixel-level features is calculated to spatially localize where to transfer the information of the discrete representation in the output space defined by the pixels of the image.

[0065] As an example, similarity may be calculated using a dot-product similarity function. Advantageously, the type of information transferred to pixel-level features is not constrained and can be any value. In particular, for depth estimation in images, depth values ​​are not explicitly processed by the internal scene discretization module. As a result, the discrete representation is free to encompass any general higher-order concepts, such as object-ness, relative positioning, and geometric structure. This is in stark contrast to existing methods that explicitly constrain what is encompassed by the discrete representation, i.e., scalar depth values.

[0066] According to an embodiment, the internal discretization module may comprise one or more cross-attention layers applied to the discrete representation (used as a query) and pixel-level features. Further, the implementation of the one or more cross-attention layers may include applying a softmax function to the discrete representation.

[0067] Specifically, the (unprojected) predicted output D obtained after layer number i can be expressed using a normal (non-transposed) softmax function as follows:

number

number

number

number

[0068] In Equation 5, the term

number

number

[0069] S50 (unprojected) Project predicted output to physical values In step S50, the (unprojected) predicted output is projected to a physical value resulting from the estimation task. In particular, if the input data is an image and the estimation task is a monocular depth estimation task, the predicted output obtained after step S40 from the internal discretization module 160 is projected to a depth value. Thus, each element of the pixel-level features is assigned a depth value, thereby providing a depth map.

[0070] The projection may be implemented by a trainable neural network layer, such as a convolutional layer, for example.

[0071] Optimization of the method can be implemented iteratively by evaluating the performance of the predicted output, which can be evaluated, for example, using the scale-invariant log-scale error defined in [Reference 15].

[0072] The architecture of an embodiment of the method for dense regression tasks is shown in Figure 2. An image is provided to an encoder 110, where the image is encoded into multiple feature maps F of various resolutions. l In the encoding feature map 112 of the plurality of encoding feature maps, the image features 111 are incorporated in a common dimension for all resolutions.

[0073] Each of the multiple encoding feature maps is provided to a decoder 120, which extracts pixel-level features from the encoding feature map. The pixel-level features have the same structure (layout) as the input data. For example, if the input data is an image, the pixel-level features may be represented as a 2D grid.

[0074] The encoded feature maps and pixel-level features are provided to an internal discretization module 160, which comprises a first discretization module 140 (e.g., an adaptive feature partitioning (AFP) module) and a second discretization module 150 (e.g., a scene discretization module). The encoded feature maps are then converted into a discrete representation (H l ) is processed by a first discretization module so that the discrete representation can be obtained by self-competition rather than hard operators, and therefore the partition is a quasi-exclusive soft partition.

[0075] A second discretization module completes the image discretization by mapping the discrete representation onto a continuous output space defined by the elements (pixels) of the pixel-level features. Due to the typical large number of elements of pixel-level features, the output space can be considered continuous. The discretization module thereby generates a predicted output.

[0076] In the architecture depicted in FIG. 2, the coded feature maps and pixel-level features of different resolutions are processed independently by instances (130, 131, 132) of the internal discretization module 160, thereby generating multiple predicted outputs (H l ) yields multiple such prediction outputs (D l ) are combined to provide the final predicted output D, taking into account all the different resolutions.

[0077] 3A-3B show the results of an embodiment of the present regression method when applied to monocular depth estimation in images from a known dataset, and a comparison with known methods.

[0078] Figure 3A shows the results of our regression method and a known method, where the method was first trained on the KITTI dataset and then tested on the official KITTI private test set. The KITTI dataset comprises stereo images of outdoor scenes captured from a moving vehicle and corresponding Velodyne LiDAR scans (see, e.g., [Reference 7]). The RGB and depth images have an average resolution of 1241 x 376. In Figure 3A, the known methods compared to our method are: PAP (see [Reference 8]), P3Depth (see [Reference 9]), VNL (see [Reference 10]), DORN (see [Reference 11]), BTS (see [Reference 3]), PWA (see [Reference 12]), ViP-DeepLab (see [Reference 13]), NeWCRF (see [Reference 14]), and PixelFormer (see [Reference 15]).

[0079] Referring to Figure 3A, the performance of the method on the KITTI dataset is evaluated based on the following criteria: scale-invariant error (SI log ), mean squared relative error (Sq.Rel), absolute mean relative error (A.Rel), and inverse depth root mean squared error (iRMS). Such metrics are often used to characterize the performance of depth estimation and are known to those skilled in the art, see for example [Reference 16].

[0080] A method is considered to provide better performance when these criteria exhibit lower values, as indicated by sentences that are lower than the criteria. Thus, Figure 3A shows that the present method is particularly efficient on the KITTI benchmark dataset, outperforming the prior art methods with which it is compared.

[0081] With reference to Figure 3B, a pruning study of our method is presented, which shows the relative efficiency of different components of our architecture (ISD: Inner Scene Discretization module, AFP: Adaptive Feature Segmentation module, MSDA: Multi-scale Deformable Attention module) relative to a component of prior art methods (EDD: Explicit Depth Discretization module).

[0082] Each row corresponds to a specific situation in which the architecture of the method includes some or none of the components. A check mark indicates that the component is included in the method, and a cross mark indicates that the component is not included in the method. For the AFP module, a check mark with the subscript "R" indicates that random initialization of the discrete representation is used, while a simple check mark (without a subscript) indicates that learnable prior knowledge is used for initialization. The EDD module and the ISD module correspond to two different ways of discretizing input data and therefore cannot be jointly used in embodiments.

[0083] The first row is the reference baseline, where, for comparison, the depth estimation task is performed without any module of the present method and without the prior art EDD module.

[0084] The performance of different architectures of the method is evaluated with respect to the following criteria: fraction of inlier pixels (δ1), root mean square error (RMS), absolute mean relative error (A.Rel). Up / down arrows next to the criteria indicate that a method is considered to perform better if the value of the criterion is higher / lower, respectively.

[0085] From the results shown in Figures 3A and 3B, we can see that the best results are obtained when the three modules of our method, namely, the ISD module, the AFP module, and the MSDA module, are used synergistically (lines 7 and 8). The performance is further improved when the AFP module is initialized with learnable prior knowledge (line 8) rather than being initialized randomly.

[0086] Furthermore, from row 2 versus rows 3-6, it can be seen that embodiments of the present method that include only one or two of the three aforementioned modules (ISD, APF, MSDA) are still more efficient than prior art methods that use only EDD modules (e.g., the methods used in [Reference 1] and [Reference 2]).

[0087] In particular, it can be seen that the ISD module alone provides a clear improvement over its EDD module counterpart with respect to the prior art improvement (row 3 vs. row 2).

[0088] From lines 4, 6, and 8, we can see that the AFP module and the MSDA module are complementary because they exhibit synergistic effects when combined. The complementarity comes from the fact that the MSDA module pre-refines the encoding feature map to be segmented, while the AFP module allows the discrete representation to adapt itself to segment the encoding feature map.

[0089] While this disclosure refers to certain preferred embodiments, modifications to those examples may be provided without departing from the overall scope of the invention as defined by the claims. In particular, individual features of the various embodiments shown / described may be combined in additional embodiments. Accordingly, the description and drawings should be regarded in an illustrative, rather than a restrictive, sense.

Claims

1. 1. A computer-implemented method for a dense regression task on input data, the computer-implemented method comprising: - determining at least one encoding feature map of said input data; - discretizing said at least one encoded feature map to obtain a discrete representation configured to represent an internal spatial pattern of said input data; - extracting pixel-level features for each element of said at least one encoded feature map; - computing a predicted output by transferring a portion of said discrete representation to the location of said pixel-level features; wherein the portion and the location are calculated based on similarity between elements of the pixel-level features and elements of the discrete representation; A computer-implemented method, wherein the dense regression task is a monocular depth estimation task in an image, and the predicted output is projected to depth values ​​to obtain a depth map.

2. The method of claim 1, wherein the steps of determining the coding feature map, obtaining the discrete representation, extracting the pixel-level features, and calculating the predicted output of the method are performed for a plurality of coding feature maps, each of the plurality of coding feature maps having a resolution parameter corresponding to a resolution of the input data so as to obtain a plurality of predicted outputs, and the method further comprises combining the plurality of predicted outputs obtained from the plurality of coding feature maps having different resolutions.

3. The method of claim 2 , wherein combining the plurality of predicted outputs obtained from the plurality of encoding feature maps having different resolutions comprises calculating an average value of the plurality of predicted outputs.

4. 4. The method of claim 2 or 3, wherein the step of determining at least one encoding feature map of the input data comprises using a multi-scale deformable attention module to refine the encoding feature maps at different resolutions.

5. 4. The method of claim 1, wherein discretizing the at least one encoding feature map to obtain a discrete representation configured to represent internal spatial patterns of the input data comprises using a cross-attention module.

6. The method of claim 5 , wherein the cross-attention module is a transposed cross-attention module configured to encourage soft clustering of the encoding feature maps between the discrete representations.

7. 7. The method of claim 6, wherein the transposed cross-attention module comprises learnable prior knowledge, and the discrete representations are iteratively obtained using the transposed cross-attention module initialized with the learnable prior knowledge.

8. The method of claim 1 , wherein the step of computing a predicted output by transferring a portion of the discrete representation to the location of the pixel-level features comprises using a cross-attention module.

9. The method of claim 1 , further comprising projecting the predicted output to a physical value.

10. A method described in any one of claims 1 to 3, wherein the depth map is an image having pixels each associated with a depth value corresponding to the estimated depth of the geometric location imaged by the corresponding pixel in the input image.

11. 1. A system for dense regression tasks on input data, the system comprising: one or more processors; and a computer-readable medium, the computer-readable medium having computer instructions stored therein, the computer instructions, when executed by the one or more processors, causing the one or more processors to: - determining, using an encoder module, at least one encoding feature map of said input data; - discretizing, using a first discretization module, said at least one encoded feature map to obtain a discrete representation configured to represent an internal spatial pattern of said input data; - using a decoder module to extract pixel-level features from said at least one encoded feature map; - transferring a portion of said discrete representation to a location of said pixel-level features using a second discretization module, said portion and said location being calculated based on similarity between elements of said pixel-level features and elements of said discrete representation, thereby calculating a predicted output; Execute The system, wherein the dense regression task is a monocular depth estimation task in an image, and the predicted output is projected to depth values ​​to obtain a depth map.

12. A computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 3 when the computer program is executed by the computer.

13. A computer readable recording medium having recorded thereon a computer program comprising instructions for performing the steps of the method according to any one of claims 1 to 3.

Images