Method of depth estimation for variable focus camera

Abstract

The invention relates to a method of depth estimation for a variable focus camera. The invention relates to a method of extracting depth information from a plurality of images taken by a camera at different focus positions, the method comprising: capturing (800) a sequence of images of a scene with the camera at different focus positions according to a predetermined focus plan, the predetermined focus plan specifying a temporal order of focus positions of the camera; extracting (802) image features of a predetermined number of captured images by a machine learning algorithm comprising a convolutional neural network, and storing the extracted image features; after extracting and storing image features from the predetermined number of captured images, processing (803) a captured image for which image features have not yet been extracted by the machine learning algorithm, the captured image representing a currently processed image, the processing comprising extracting (804) image features from the currently processed image by the machine learning algorithm and storing the extracted image features, the processing further comprising: aligning (805) image features stored from previously captured images with image features of the currently processed image; and generating (806) at least one multi-dimensional tensor representing image features of at least some of the processed images aligned with image features of the currently processed image; and generating (807) a two-dimensional depth map using focus positions specified in the predetermined focus plan and the at least one generated multi-dimensional tensor.

Description

Technical Field

[0001] The present invention relates to a method of the type specified in the preamble of claim 1, a computer system specified in the preamble of claim 13, and a storage medium specified in the preamble of claim 15. Background Technology

[0002] When capturing a 2D image from a 3D scene, such as when capturing an image from a scene in actual physical 3D space, depth information is lost when the 3D scene is projected onto the 2D image.

[0003] An important and challenging task is to recover at least part of the lost spatial depth information in computer vision or robot vision, especially for autonomous robot systems, so as to be able to, for example, identify three-dimensional objects and / or distinguish background and foreground features or objects in captured two-dimensional images.

[0004] Existing technologies require the use of specialized, complex, and specific hardware, such as stereo cameras and / or complex optical microlens arrays, and / or require complex and computer-intensive processing of the captured images, such as structural features from mobile technologies and / or depth features from focusing technologies.

[0005] In particular, current techniques such as those based on structure from mobile technologies and / or depth from focusing technologies produce poor results when attempting to extract or recover depth information from dynamic scenes (e.g., when trying to extract depth information from an image stream), where movement of objects or features occurs simultaneously with the image stream capturing the scene. Summary of the Invention

[0006] Technical issues

[0007] Another object of the present invention is to provide an improved apparatus for extracting / estimating / recovering depth information from two-dimensional images captured by a camera / image capture device from an actual three-dimensional scene. In particular, for example, the objective of the present invention is to accelerate the process of estimating depth information from multiple captured images from a scene and to improve the accuracy of the estimated depth information, especially in the case of capturing images from a dynamic scene.

[0008] Technical solution

[0009] According to the present invention, the objective is achieved by a computer-implemented method, computer system, and computer storage medium according to the independent claims.

[0010] Advantageous embodiments and further developments are the subject of the dependent claims.

[0011] For example, a computer-implementable method for extracting depth information from multiple images taken by a camera at different focal positions may include one, some, or all of the following steps.

[0012] • Capture a sequence of images of the scene at different focus positions using the camera, according to a predetermined focusing plan that specifies the time sequence of the camera's focus positions.

[0013] • Extract a predetermined number of image features from the captured images using a machine learning algorithm that includes a convolutional neural network or a convolutional neural network, and store the extracted image features.

[0014] After extracting and storing image features from the predetermined number of captured images, captured images whose image features have not yet been extracted are processed using a machine learning algorithm. These captured images represent the images currently being processed.

[0015] The processing also includes aligning image features stored from previously captured images with image features of the currently processed image, and

[0016] Generate at least one multidimensional tensor, which represents image features of at least some processed images (preferably all processed images) aligned with the image features of the currently processed image.

[0017] • And a two-dimensional depth map is generated using the focal position specified in the predefined focusing plan and at least one generated multidimensional tensor.

[0018] Here, the expression for at least some processed images can refer to a subset of a predetermined number of captured images after processing (i.e., whose image features have been extracted and stored), or it can refer to a set including a predetermined number of captured images from all processes. Furthermore, the expression for all processed images can refer to a set including a predetermined number of captured images from all processes, or it can refer to a set including a predetermined number of captured images from all processes and the currently processed image.

[0019] For example, if k is a natural number representing a predetermined number of captured images (which may include images currently being processed), then the representation of at least some of the processed images may refer to a natural number s, where s is less than or equal to k and where s is greater than or equal to 2. Preferably, s may be equal to k.

[0020] Here, "scene" can be understood as a scene in actual physical three-dimensional space.

[0021] Here, an image or image frame can be understood as a two-dimensional pixelated digital image or image frame with width and height dimensions.

[0022] Here, one or more images or image frames can be images / image frames from a video stream.

[0023] Here, image features can be understood in particular as the characteristics or attributes of objects or subjects in the captured image, such as the shape, outline, and color of objects or subjects in the captured image. Image features can also refer to information describing whether an object or subject is in focus or out of focus; that is, image features can include focus / out-of-focus information.

[0024] The aforementioned predefined focus plan can specify the order in which the camera captures images at specific focus positions at specific times. In other words, the focus plan can specify a predefined list of focus positions to be used by the camera, and can also specify how to traverse that list when capturing a sequence of images of a scene with the camera.

[0025] The predetermined focusing plan can include multiple focus positions, which may include different focus positions and / or the same focus position. In other words, the focusing plan can specify that the same focus position of the camera can be used to capture images at different points in time.

[0026] The camera can then iterate through the exemplary focusing scheme in chronological order to capture images at the defined focus locations.

[0027] Here, "camera" can be understood as an image capture system capable of capturing pixelated two-dimensional digital images. Furthermore, it should be understood that the exemplary camera can capture image streams, such as video streams of images. Specifically, "camera" here can be understood as an optical camera.

[0028] An exemplary camera can be a regular digital camera, such as the camera of a smartphone. Furthermore, the camera can be configured to capture images at different focus positions, i.e., to traverse a predetermined focus plan.

[0029] For example, a camera may include optical elements, such as one or more lenses, for controlling where captured light converges on an image detection plane, such as an image sensor plane; that is, the camera may be a zoom camera.

[0030] The exemplary method steps described above can be performed iteratively or cyclically until all captured images or until the desired number of captured images are processed.

[0031] The methods described above and herein for extracting depth information from multiple images captured by a camera at different focus positions enable the extraction of depth information from images captured at different focus positions faster and more accurately than current out-of-focus depth algorithms.

[0032] The accelerated extraction of depth information from images focused at different locations, according to the method steps described herein, allows for real-time extraction of depth information from an image stream (i.e., without delay perceptible to the camera user).

[0033] For example, depth information can be extracted simultaneously from images in the captured video stream during camera capture. For instance, depth information can be extracted at a rate of less than 18 ms per image, thus allowing the methods described herein to be applied, for example, to video streams with image frame rates of at least 30 frames per second or higher.

[0034] In addition, for example, before the user of the camera is committed to taking a picture of the scene, the camera may automatically capture multiple images of the scene at different focal lengths according to a predetermined focusing plan, in order to extract depth information for the scene in order to optimize the focus settings of the image that the user wants to capture and / or generate a panoramic focus image of the scene that the user wants to capture.

[0035] The improved performance of the method described herein for extracting depth information from multiple images is due in particular to the fact that, when processing the current image or the current input image, information from at least some or all of the previously captured / past images is preserved, reused, and considered, thereby avoiding the need for additional computation.

[0036] Furthermore, because the method described herein for extracting depth information from multiple images does not require any complex mathematical operations, but rather relies on basic mathematical operations or calculations such as addition and multiplication that can be performed in parallel, it is well-suited for execution by a graphics processing unit (GPU), which can perform such basic parallel calculations faster than a general-purpose central processing unit (CPU).

[0037] Furthermore, the method for extracting depth information from multiple images described herein provides more robust and accurate extraction of absolute depth information from images of a scene, particularly in dynamic scenes (i.e., scenes in which movement of objects and / or subjects occurs during the capture of multiple images).

[0038] The improved robustness and accuracy of the method described herein are due in particular to the fact that the processing of the captured image is performed only at the feature level rather than at the level of the image or image frame as a whole.

[0039] The method described herein eliminates the need to directly register images / image frames relative to each other, which is required by common depth-of-focus techniques and causes particular problems when objects or subjects in a scene move between captured images, where such movement can cause current techniques for image alignment to fail.

[0040] Compared to current technologies, the method described herein allows alignment at the feature level, that is, aligning past / previously captured features from past / previously captured images with image features from the currently processed image / currently processed input image, thereby providing indirect implicit registration / alignment between captured images. Therefore, the method of the present invention is better able to handle movement in the scene and / or movement caused by camera movement (e.g., due to camera support wobbling that may occur between captured images).

[0041] In addition to using the extracted depth information, i.e. the generated two-dimensional depth map, to generate a better focused image, the methods and techniques described here can also be applied to the fields of computer and robot vision.

[0042] For example, the methods described herein can be used to improve visual object recognition in robots or vehicles (e.g., autonomous robots or vehicles), and in particular to enhance autonomous navigation capabilities.

[0043] The methods described herein can be used to improve image classification, such as improving the accuracy and performance of image or object detection algorithms, and / or to improve image / video analysis systems using extracted depth information.

[0044] Furthermore, the extracted depth information, i.e. the generated two-dimensional depth map, can be used as input for a display, particularly for example, as input for a three-dimensional display, to generate a three-dimensional image of a scene captured in one or more two-dimensional images.

[0045] Furthermore, the methods described herein and the machine learning algorithms including convolutional neural networks described herein are independent of any type of camera used; that is, they are applicable to any type of image and camera. No specific camera hardware or lens is required.

[0046] Convolutional neural networks can be trained with different camera and focus configurations, allowing the method to be better generalized to unknown scenes / unknown samples.

[0047] In the method steps described above and herein, image features may be extracted by layers of a convolutional neural network into a three-dimensional feature tensor comprising a width dimension W, a height dimension H, and a channel dimension C, wherein the channel dimension may describe the number of feature maps extracted from the image through one or more layers of the convolutional neural network, and wherein the storage of the extracted image features may include a list of the extracted image features stored as three-dimensional feature tensors.

[0048] An exemplary feature tensor, such as the feature tensor F of the currently processed image, can therefore be One of the elements, namely C, W, and H refer to the dimensions mentioned above.

[0049] Here, the feature map extracted from the image can be understood as a two-dimensional representation of the features, aspects, attributes, or characteristics of the image detected / extracted by one or more layers of a convolutional neural network, having a width dimension W and a height dimension H. Specifically, the feature map can be understood as a two-dimensional representation of location, where specific features, aspects, attributes, or characteristics of the image are detected or extracted by one or more layers of a convolutional neural network.

[0050] The width and height dimensions of the feature map may be equal to or different from (e.g., less than) the width and height dimensions of the image from which the feature map is extracted.

[0051] The exemplary feature map can be understood as a slice representing the exemplary three-dimensional feature tensor.

[0052] Here, the term "image features" or "image characteristics" can therefore be understood as a representation of an image or a representation of aspects of an image in a different dimensional space (e.g., a higher dimensional space) than the dimensional space from which the features are extracted.

[0053] Therefore, the process described here of extracting image features from an image using machine learning algorithms including convolutional neural networks can be understood as converting an image into image features.

[0054] Representing the extracted image features as tensors facilitates computational processing of these features, as feature tensors can be more easily processed by convolutional neural networks.

[0055] Convolutional neural networks can extract image features through a sequence of operations including convolutional layers, batch normalization (BN), rectified linear activation function (ReLU), resampling (e.g., upsampling), shaping or pooling, concatenation, and skipping operations.

[0056] Here, batch normalization (BN) can refer to a normalization operation using an estimated mean E(x) (also known as the running mean) and variance Var(x), along with a scale parameter and a shift parameter, which may have been learned by the convolutional neural network during training.

[0057] Therefore, batch normalization (BN) can be expressed as follows:

[0058]

[0059] Here, the Corrected Linear Activation Function (ReLU) can be understood as the activation function of a convolutional neural network, which sets negative values ​​to zero and keeps positive values, including zero, unchanged.

[0060] Therefore, the corrected linear activation function (ReLU) can be expressed, for example, as:

[0061] ReLU(x) = max(x, 0) (b)

[0062] Here, the upsampling operation can refer to the operation of increasing the width and / or height dimensions of an image, image features, image feature tensors, or feature maps, where the upsampling operation is based on / includes interpolation or extrapolation of the values ​​of the original image, original image feature tensors, or original feature maps to obtain the upsampled value.

[0063] Here, reshaping can refer to modifying the shape (i.e., dimension) of a tensor (e.g., the shape of an image, image feature, feature map, or image feature tensor) while maintaining the total number of elements in the tensor. For example, a tensor of shape [10,3,9,2] can be reshaped to [10,3,18], [30,18], [30,1,18], or

[540] , because all these shapes contain the same total number of elements (540).

[0064] Here, pooling operations (pooling, pooling, spatial pyramid pooling) can be understood as operations similar to upsampling operations but downsampling the width and / or height dimensions of an image, image features, image feature tensor, or feature map. For example, a pooling operation can apply an operation, such as a max or average function, to a subset of pixels, such as pixels of an image, image features, feature map, or image feature tensor, where the subset corresponds to a predetermined kernel / filter size with a predetermined stride to generate a pixel.

[0065] For example, a max pooling operation with kernel size = 2 and stride = 2 downsamples the input (e.g., an image, image feature, image feature tensor, or feature map) by half using a subset of pixels of shape [2,2], where the downsampling factor corresponds to the specified stride.

[0066] The selected kernel and stride size are merely examples. Other kernel and stride sizes can also be selected, for example, based on experience.

[0067] Here, a cascading operation can refer to the operation of merging two or more tensors (e.g., images, image features, image feature tensors, or feature maps) along a specific dimension. This operation may require all tensors to be merged / cascaded to have the same shape in all dimensions except the cascading dimension.

[0068] For example, cascading tensor t1 of shape [10,3,9] and tensor t2 of shape [4,3,9] along the first dimension will produce tensor of shape [14,3,9].

[0069] Here, skipping operations can refer to operations that connect discontinuous layers or discontinuous operation sequences in a convolutional neural network using specific operations (such as addition operations).

[0070] For example, let x1 be the output of the operation in layer 1, where x0 is the input, for example x1 = layer1(x0). Let x2 be the output of the operation in layer 2, where x1 is the input, for example x2 = layer2(x1), where layer 2 is continuous with layer 1. Let x3 be the output of the operation in layer 3, where x2 is the input, for example x3 = layer3(x2), where layer 3 is continuous with layer 2.

[0071] Then, an exemplary skip operation can be defined as the sum of x3 and x1, i.e., skipping the second layer.

[0072] Here, a convolutional layer can refer to a convolution operation on an image, image features, image feature tensors, or feature maps.

[0073] The possible exemplary operations or layers of a convolutional neural network can be performed along multiple dimensions / on multiple dimensions / in multiple dimensions. The dimension of the operation may depend on where the operation occurs in the convolutional neural network.

[0074] For example, the extraction of image features by a convolutional neural network may involve operations or layers that specifically operate on the height (H) and width (W) dimensions of the image, image features, or image feature tensors or feature maps, and further processing of the extracted image features, such as the alignment of image features, may involve operations or layers of the convolutional neural network that operate on further dimensions, such as the focus position dimension.

[0075] The following provides further examples of specific configurations of exemplary convolutional neural network architectures configured for extracting image features.

[0076] The alignment described above between image features stored from a previously captured image and image features from the currently processed image may include applying four-dimensional encoding to image features stored from a previously captured image and image features from the currently processed image, wherein the image features are represented as tensors.

[0077] For example, the four-dimensional encoding may include embedding temporal, spatial, and focal position information into image features from previously captured images as well as embedding it into image features from currently processed images.

[0078] For example, the four-dimensional code E can be composed of the following two equations:

[0079]

[0080]

[0081] Where α is a correction constant, preferably greater than C, the number of channels or channel dimension size x, y are spatial pixel coordinates, t is time, i.e. the time position / time point / time stamp / time index of the captured image from which image features are extracted, where t∈[0, K-1], where K represents the number of previously captured images, such as a predetermined number of captured images, d∈[0, N-1] is the focal plane position / focus position / focus position index of the given image to be encoded, and N is the total number of images, such as the number of images in the focus plan or the sum of the number of images in the focus plan and the number of images derived from the images in the focus plan, wherein the derived images can be derived by interpolation or extrapolation of the images captured according to the focus plan, and i∈[0, C / 2] is the index used to divide the number of channels into even and odd channels for encoding.

[0082] Encoded by example E 2i，x，y E 2i+1，x，y The exemplary encoding E is applied to the image features / feature tensors of the currently processed image through addition. And it is applied to each of the image features / feature tensors from the previously captured images, that is, each of the image features / feature tensors from the past K images. Obtained as follows and

[0083] EF c，x，y =F c，x，y +E c，x，y (3)

[0084] EPF k，c，x，y =PF k，c，x，y +E k c，x，y (4)

[0085] Where c∈[1,C] is the channel index, E k c，x，y The encoding of the image features / feature tensors of the past K images / previously captured images is represented by k∈[1,K], which represents the index of the image features / feature tensors used for the past K images / previously captured / stored images.

[0086] The example above describes an exemplary four-dimensional encoding that is non-linear and based on the use of trigonometric functions, wherein the four-dimensional encoding is applied by addition to image features from the currently processed image, and to each image feature stored from previously captured images. However, other four-dimensional encodings may also be used.

[0087] To align the encoded feature tensors EF and EPF and generate, for example, at least one multidimensional tensor representing the image features of all previously processed images of K processed images aligned with the image features of the currently processed image, the following exemplary steps can be performed.

[0088] Convolutional neural networks can perform similarity operations on encoded feature tensors based on the following exemplary similarity scores:

[0089]

[0090] Here, EF' is a two-dimensional matrix obtained by shaping EF using a convolutional neural network, where i and j represent matrix elements. EPF' is a three-dimensional tensor obtained by shaping EPF using a convolutional neural network, where k, i', and j represent elements of the three-dimensional tensor, and k is the index of the image feature tensor representing the past K / previously captured / stored images. For example, indices i and i' have a range of [0, (H*W)-1], and index j can have a range of [0, C-1], where H and W are the height and width dimensions of the feature tensor, and C is the number of channels.

[0091] For completeness, as mentioned earlier, note that the EPF may also contain features extracted from the currently processed image, i.e., it may contain the feature tensor of the currently processed image. However, since the feature tensor of the currently processed image must be perfectly similar to itself, the similarity score is not affected when the feature tensor of the currently processed image is included in the EPF.

[0092] However, including the feature tensors of the currently processed image in the EPF can be particularly useful for checking the effectiveness and robustness of convolutional neural networks, especially during the training of convolutional neural networks.

[0093] therefore, This can be understood as the similarity score between the image features of the currently processed image and the image features of each of the K past images.

[0094] According to the following formula, the similarity score can be converted into a probability.

[0095]

[0096] in It is a normalized similarity score with the following properties:

[0097]

[0098] The exemplary normalized similarity score can then be compared with the past K images. Multiplying the integer encoded feature tensors to obtain

[0099]

[0100] Then AF′ can be shaped into Then, AF can be grouped along the first dimension to group features corresponding to the same focal position, thereby obtaining in Where K is the total number of the past K / previously captured / stored images (which may also include the currently processed images) or the total number of all focus positions of the past K / previously captured / stored images (which may also include the currently processed focus position images), and N is the number of unique focus positions among the total number of focus positions K.

[0101] Then, the information can be combined, for example, through a reduction sum operation:

[0102]

[0103] by This is an example of at least one multidimensional tensor representing the image features of all processed images aligned with the image features of the currently processed image, where n is an index in the range [0, N-1] and m is an index in the range [0, M-1], where N is the number of unique focal positions and M is as described above.

[0104] As mentioned earlier, it's also possible that instead of aligning all stored or processed image features from previously captured images with the image features of the currently processed image, it's conceivable that only some image features from a predetermined number of captured / past K images are aligned with the image features of the currently processed image. In other words, it's possible, for example, It may only represent some previously processed image features that are aligned with the image features of the currently processed image; that is, the above index range is merely exemplary.

[0105] The step of generating a two-dimensional depth map using a focal location specified in a predetermined focusing plan and at least one generated multidimensional tensor (EPF) may further include: using the obtained at least one multidimensional tensor EPF via a machine learning algorithm. a Generate at least one multidimensional focusing probability map The at least one multidimensional focus probability map is then remapped to the actual physical distance using the focus location specified in the predetermined focus plan.

[0106] The multidimensional focusing probability map fpm can be obtained, for example, through a convolutional neural network via the following steps:

[0107] • If necessary, the EPF can be optionally added. a Upsampled to the original spatial resolution of the captured image during processing

[0108] • Convolution to reduce the number of channels to 1

[0109] ·Use the softmax operation to normalize and obtain a multidimensional focusing probability map.

[0110] Here, the softmax operation of a convolutional neural network can be defined, for example, as follows:

[0111]

[0112] And has the attribute ∑ i smax(x) i =1 and smax(x) i ∈[0,1].

[0113] The obtained exemplary at least one multidimensional focusing probability map It is a three-dimensional tensor with a width dimension W, a height dimension H, and a focal position dimension N, wherein the focal position dimension describes the number of focal positions, such as different focal positions in the focus plan or different focal positions from the focus plan and from the composite / derived image, which has been derived from the captured image by interpolation or extrapolation.

[0114] Here, the size of the width and height dimensions can be equal to the size of the width and height dimensions of the input image, wherein the input image is either an image of a predetermined number of captured images, an image currently being processed, or a synthesized image.

[0115] Remapping at least one multidimensional focus probability map (fpm) to an actual physical distance using the focus locations specified in a predefined focus plan may include calculating the dot product between each pixel of the at least one multidimensional focus probability map and a known focus location in the focus plan, thereby obtaining a two-dimensional depth map with absolute depth information about the captured scene.

[0116] The steps described above can be performed for different image feature scales. Specifically, the steps of extracting image features from a predetermined number of captured images and extracting image features from the currently processed image can further include extracting image features at different scales using a machine learning algorithm, wherein the scale is defined as a fraction of the height of the input image and / or a fraction of the width of the input image, wherein the input image is an image of the predetermined number of captured images or the currently processed image.

[0117] In addition, image features / feature tensors extracted from a predetermined number of captured images and image features / feature tensors extracted from the currently processed image are stored in a circular buffer in a computer-readable memory. For example, a circular buffer containing image features from at least a predetermined number of captured images can be stored.

[0118] This allows for efficient memory management and efficient processing of image streams, such as those from video streams, while avoiding memory overflow.

[0119] Here, the predetermined number of images to be captured can be at least equal to or greater than the number of different focal points specified by the focusing plan.

[0120] The convolutional neural network described above and herein can be a trained convolutional neural network trained on training samples, which include multiple images focused at different focal points for multiple different scenes from the real physical world, where the scenes are static or dynamic, and where the parameters of the convolutional neural network have been optimized by comparing the estimated depth map generated by the convolutional neural network with the corresponding known ground truth depth map (i.e., the depth map whose absolute value is known) using a loss function.

[0121] The loss function is a measure of the difference between the estimated / predicted depth map and the expected known ground truth depth map. The convolutional neural network (CNN) is trained until the loss function reaches its expected / specified minimum and the optimal model parameters of the CNN are determined. Minimizing the loss function can be achieved using optimization techniques (e.g., gradient descent). However, other optimization techniques, such as simulated annealing, genetic algorithms, or Markov chain Monte Carlo algorithms, can also be applied to minimize the loss function and determine the optimal model parameters of the CNN from the training, such as the optimal weights, scale, or shift parameter values ​​for convolutional layers.

[0122] The methods and method steps described above and herein can be implemented on a computer system including: computer memory, one or more processors (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)), wherein the computer memory can store instructions that direct the one or more processors to perform the methods or method steps described herein for extracting depth information from multiple images captured by a camera at different focal positions.

[0123] Because the method steps described herein enable faster image processing compared to known defocus depth techniques, real-time processing of images and image streams can even be achieved on the processors of portable devices such as smartphones.

[0124] Therefore, the computing system can be a portable mobile device, such as a smartphone, including a camera configured to capture images of a scene with different focus positions.

[0125] Furthermore, computer-executable instructions that, when executed by a computer system, can perform a method for extracting depth information from multiple images captured by a camera at different focal positions as described herein, can be stored on a computer-readable storage medium, such as a non-volatile computer storage medium.

[0126] The predetermined number of captured images may be less than, equal to, or greater than the number of focal positions in the predetermined focusing plan and / or may be equal to or greater than the number of different, i.e., unique focal positions in the predetermined focusing plan. In particular, the predetermined number of captured images may be a natural multiple of the number of focal positions in the predetermined focusing plan. Attached Figure Description

[0127] The following figure illustrates this example:

[0128] Figure 1 Exemplary schematic overview of methods and apparatus for extracting depth information

[0129] Figure 2a An exemplary two-dimensional encoder of a convolutional neural network

[0130] Figure 2b Example 2D convolutional block

[0131] Figure 2c Example 2D residual convolution block

[0132] Figure 2d : Example of a two-dimensional multi-scale feature aggregation block

[0133] Figure 2e : Exemplary two-dimensional spatial pyramid pooling block

[0134] Figure 3a An exemplary 3D decoder of a convolutional neural network

[0135] Figure 3b Example 3D residual convolution block

[0136] Figure 3c Example 3D multi-scale feature aggregation block

[0137] Figure 3d : Exemplary 3D spatial pyramid pooling block

[0138] Figure 4a Example memory block

[0139] Figure 4b Example feature alignment block

[0140] Figure 4c Example feature alignment header

[0141] Figure 5 Example flowchart of a method for extracting depth information

[0142] Figure 6 An illustrative overview of training examples for machine learning algorithms. Detailed Implementation

[0143] Figure 1 An exemplary general overview of methods and apparatus for extracting depth information from images is shown.

[0144] The scene's image stream 700 (which has been captured by a camera with a variable focal length by capturing images at different focal positions according to a focus plan 710) is input / feeded to a machine learning model / machine learning algorithm 720 (including a convolutional neural network).

[0145] The machine learning algorithm, including a convolutional neural network, outputs a focus probability map 730 of the scene, which can be remapped 740 to an absolute distance using the known focus position of the focus plan 710 to obtain a two-dimensional depth map 750 of the scene.

[0146] Figure 2a An exemplary portion of an exemplary possible convolutional neural network architecture is shown, which can be used to extract image features from image 101, which has been captured by a camera at different focal positions, and output exemplary extracted features or feature tensors 115, 116, 117 and 118.

[0147] Figure 2a An exemplary portion of the exemplary possible convolutional neural network architecture shown can be understood as representing a two-dimensional (2D) encoder 100 that encodes features from the input image 101 into a two-dimensional feature map with a width of W and a height of H for each channel dimension C.

[0148] In other words, the image features are extracted as three-dimensional feature tensors 115, 116, 117, 118, including a width dimension W, a height dimension H, and a channel dimension C, wherein the channel dimension describes the number of feature maps extracted from the image through one or more layers or blocks 102, 103, 104, 106, 107, 108, 109, 110, 111, 112, 113, 114 of the shown portion of the convolutional neural network.

[0149] In the exemplary case shown, features from input image 101 are extracted at four different scales, such as having different spatial sizes and / or different channel dimensions.

[0150] For example, the 3D output feature tensor / extracted feature tensor 115 can be shaped (channel dimension C = 16, H / 8, W / 8), feature tensor 116 can be shaped (C = 16, H / 16, W / 16), feature tensor 117 can be shaped (C = 16, H / 32, W / 32), and feature tensor 118 can be shaped (C = 32, H / 64, W / 64), where H and W are the height and width dimensions of the input image 101.

[0151] Note that the number and selection of different scales are merely illustrative, and a single scale may also be used. Furthermore, the number of channels is only illustrative and can be determined / defined empirically, for example.

[0152] In the following text, two-dimensional (2D) operations, layers, or blocks, such as 2D convolutional blocks, 2D residual convolutional blocks, 2D spatial pyramid pooling blocks, or 2D multi-scale feature aggregation blocks, can be understood as acting / operating on the height and width dimensions of a feature tensor, such as the height and width dimensions of a feature map. The height and width dimensions may be the same in size as or different in size from the height and width dimensions of the input image 101.

[0153] Exemplary feature extraction at four different scales is achieved through a sequence comprising a two-dimensional convolutional block 102 and four two-dimensional residual convolutional blocks 103, 104, 105, and 106. Each of the exemplary two-dimensional residual convolutional blocks 103, 104, 105, and 106 contains the following sequence: a two-dimensional convolutional layer (Conv), batch normalization (BN), a corrected linear activation function (ReLU), summation (Sum), and a skip connection between the input and output of a given residual convolutional block. Figure 2c An exemplary configuration of a two-dimensional residual convolutional block is provided in the document.

[0154] The two-dimensional convolutional block 102 may include, for example, the following sequence: a two-dimensional convolutional layer (Conv), batch normalization (BN), a corrected linear activation function (ReLU), and a pooling layer (pool). Figure 2b An exemplary configuration of two-dimensional convolutional blocks is provided in the document.

[0155] After applying the last residual convolution block 106, apply the two-dimensional spatial pyramid pooling block 107. Figure 2e An exemplary configuration of such a two-dimensional spatial pyramid pooling block is provided in the document.

[0156] Then, the output of the two-dimensional spatial pyramid pooling block 107 is sequentially merged with the intermediate outputs from the first three two-dimensional residual convolutional blocks 103, 104 and 105 using two-dimensional multi-scale feature aggregation blocks 108, 109 and 110.

[0157] Figure 2dAn exemplary configuration of a two-dimensional multi-scale feature aggregation block is provided.

[0158] As a final step, for each scale, the sequence 111, 112, 113, 114a of two-dimensional convolutional layers (Conv), batch normalization (BN) 111b, 112b, 113b, 114b and corrected linear activation functions (ReLU) 111c, 112c, 113c, 114c can be applied to obtain the extracted features / feature tensors 115, 116, 117, 118 for the four exemplary feature scales.

[0159] Figure 2b An example is shown Figure 2a A possible exemplary configuration of the two-dimensional convolutional block 102 includes three sequences 119, 120, and 121, each of which includes a two-dimensional convolutional layer (Conv), batch normalization (BN), and rectified linear activation function (ReLU) operations. After the last sequence 121, a pooling layer is applied to obtain the output of the convolutional block 102.

[0160] Figure 2c An example is shown for use Figure 2a Possible exemplary configurations of two-dimensional residual convolutional blocks 103, 104, 105, and 106, including two branches 128 and 129.

[0161] The exemplary branch 128 includes a first sequence 123 and a second sequence 124. The first sequence 123 includes a two-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations, and the second sequence 124 includes batch normalization (BN) and corrected linear activation function (ReLU) operations.

[0162] Exemplary branch 129 consists of only a single sequence of two-dimensional convolutional layers (Conv) and batch normalization (BN) operations.

[0163] The outputs of the two exemplary branches are combined using the summation (Sum) operation 125, and the output of the two-dimensional residual convolution block is obtained after the final corrected linear activation function (ReLU) operation 126.

[0164] Figure 2d An example is shown for use Figure 2a Possible exemplary configurations of two-dimensional multi-scale feature aggregation blocks 108, 109, and 110.

[0165] The exemplary two-dimensional multi-scale feature aggregation block may include an upsampling operation (UP) 130, followed by a sequence 131, which includes a two-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations, followed by a concatenation operation 132 and a final sequence 133, which includes a two-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations.

[0166] Figure 2e An example is shown Figure 2a A possible exemplary configuration of the two-dimensional spatial pyramid pooling block 107. The input of the exemplary two-dimensional spatial pyramid pooling block is directed to five branches 134, 135, 136, 137, and 138, wherein four parallel branches 134, 135, 136, and 137 each include a sequence of pooling layers, convolutional layers, and up-sample operations. The outputs of the four parallel branches 134, 135, 136, and 137 are then merged with a fifth branch 138 corresponding to the input of the two-dimensional spatial pyramid pooling block through a summation operation (Sum) 139 to generate the output of the two-dimensional spatial pyramid pooling block. That is, branch 138 skips the operations of the four parallel branches 134, 135, 136, and 137.

[0167] Figure 3a An exemplary portion of a possible convolutional neural network architecture is shown, which can be followed Figure 2a The outputs 115, 116, 117, and 118 of the exemplary encoder 100 shown, i.e., the extracted feature / feature tensors 115, 116, 117, and 118, become Figure 3a The input of the exemplary 3D decoder 200 shown.

[0168] For example, the 3D output feature tensor / extracted feature tensor 115 of the example shape (C=16, H / 8, W / 8) can become input 204, the shape (C=16, H / 16, W / 16) can become input 203, the feature tensor 117 of the shape (C=16, H / 32, W / 32) can become input 202, and the shape (C=32, H / 64, W / 64) can become input 201 of the decoder 200.

[0169] The exemplary decoder 200 outputs a final three-dimensional focus probability map 310 and three other intermediate focus probability maps 280, 290, and 300, all of which have shapes (N, H, W), where N is, for example, the number of different focal points in the focus plan, and H and W correspond to... Figure 2a The height and width dimensions of the input image 101.

[0170] However, it's conceivable that N here also represents additional focal positions not specified in the focusing plan but synthesized by the convolutional neural network. Such synthesized / generated focal positions can be used to obtain more additional focal probability maps and thus increase the achievable depth resolution.

[0171] Each of the input features / feature tensors 201, 202, 203, and 204 first passes through dedicated memory blocks 240, 250, 260, and 270, where stored features of past / previously captured and previously processed images are retrieved and aligned with features of the currently processed image (e.g., input image 101), producing a multidimensional tensor of shape (C, N, H, W), where C is the number of channels in the feature map, N is the number of different focal lengths in the focusing plan, and H and W refer to the spatial resolution of the extracted features, i.e., the height and width dimensions of the feature map. The multidimensional tensor represents, for a given scale, image features extracted from previously processed images aligned with image features extracted for the currently processed image.

[0172] Figure 4a An example of a memory block is shown.

[0173] In the following text, three-dimensional (3D) operations, layers, or blocks, such as 3D residual convolution blocks, 3D spatial pyramid pooling blocks, or 3D multi-scale feature aggregation blocks, can be understood as acting / operating on the height and width dimensions of a feature tensor, such as the height and width dimensions of a feature map, and acting / operating on the focal position dimension. The height and width dimensions may be the same in size as or different in size from the height and width dimensions of the input image 101.

[0174] After passing through memory blocks 240, 250, 260, and 270, one or more three-dimensional (3D) residual convolution blocks 320, 350, 380, and 410 can be applied. Figure 3a In this example, for a given feature scale, only one three-dimensional (3D) residual convolutional block is shown, but it can have more than one, such as five.

[0175] Examples of three-dimensional (3D) residual convolutional blocks are as follows: Figure 3b As shown.

[0176] Each of the residual convolutional blocks 320, 350, 380, and 410 is followed by a three-dimensional (3D) spatial pyramid pooling block 330, 360, 390, and 420.

[0177] Examples of three-dimensional (3D) spatial pyramid pooling blocks are as follows: Figure 3d As shown.

[0178] The output examples for pyramid pooling blocks 330, 360, and 390 follow two branches:

[0179] • A branch 430, 440, 450, in which the input image 101 is upsampled to its original spatial resolution (UP), followed by a sequence of convolutional layers (Conv), batch normalization (BN) and corrected linear activation function (ReLU), another convolutional layer (Conv) and another batch normalization (BN) operation (to reduce the number of channels to one), and finally a softmax operation (to obtain the intermediate focus probability maps 280, 290, 300).

[0180] Another branch, 431, 441, and 451, includes three-dimensional (3D) multi-scale aggregation blocks 340, 370, and 400, which merge the outputs of the three-dimensional spatial pyramid pooling blocks with the outputs of memory blocks 250, 260, and 270. Specifically, in the illustrated exemplary architecture, the output of memory block 250 is merged with the output of the three-dimensional spatial pyramid pooling block 330, the output of memory block 260 is merged with the output of the three-dimensional spatial pyramid pooling block 360, and the output of memory block 270 is merged with the output of the three-dimensional spatial pyramid pooling block 390.

[0181] Examples of 3D multi-scale aggregate blocks are as follows: Figure 3c As shown.

[0182] After aggregating features from all scales and passing through the last 3D residual convolutional block 410 and the last 3D spatial pyramid pooling block 360, the final focused probability map 310 can be obtained by applying the last sequence 460, which includes convolutional layers (Conv), batch normalization (BN) and corrected linear activation function (ReLU), another convolutional layer (Conv) and another batch normalization (BN) operation, and a final softmax operation.

[0183] Using the final focusing probability map 310fpm output by the convolutional neural network, it can be obtained through ∑ n fpm n,i,j *f n =dmap i,j Obtain a 2D depth map (dmap) i,j Where i and j are natural number indices for the height and width dimensions, and f n is a unique focal position, which can include the unique focal position of the focusing scheme as well as any other possible unique focal positions that may have been synthesized / generated by the convolutional neural network, and n is a natural number index.

[0184] Figure 3b This demonstrates what can be achieved in an exemplary convolutional neural network architecture. Figure 3a Exemplary configurations of three-dimensional residual convolutional blocks 320, 250, 380, 410 used in an exemplary three-dimensional decoder 200.

[0185] A three-dimensional residual convolution block can include two branches, 501 and 502.

[0186] The exemplary branch 501 includes a first sequence 503 and a second sequence 504. The first sequence 503 includes a three-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations. The second sequence 504 includes batch normalization (BN) and corrected linear activation function (ReLU) operations.

[0187] Exemplary branch 502 consists of only a single sequence of three-dimensional convolutional layers (Conv) and batch normalization (BN) operations.

[0188] The outputs of the two exemplary branches are combined using the summation (Sum) operation 506, and the output of the three-dimensional residual convolution block is obtained after the final corrected linear activation function (ReLU) operation 507.

[0189] Figure 3c It shows Figure 3a Possible exemplary configurations of three-dimensional multi-scale feature aggregation blocks 340, 370, and 400.

[0190] The exemplary three-dimensional multi-scale feature aggregation block may include an upsampling operation (UP) 508, followed by a sequence 509, which includes a three-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations, followed by a concatenation operation 510 and a final sequence 511 including a three-dimensional convolutional layer (Conv), batch normalization (BN), and corrected linear activation function (ReLU) operations.

[0191] It is conceivable that the aforementioned possible synthetic focal points can be generated within a three-dimensional multi-scale feature aggregation block. For example, a three-dimensional upsampling operation can be used to generate synthetic focal points before the concat operation 510.

[0192] Figure 3d An example is shown Figure 3a Possible exemplary configurations of three-dimensional spatial pyramid pooling blocks 330, 360, 390, and 420.

[0193] The input to the exemplary 3D spatial pyramid pooling block is directed to five branches 512, 513, 514, 515, and 516. Each of the four parallel branches 512, 513, 514, and 515 includes a sequence of pooling layers, convolutional layers, and upsampling operations. The outputs of the four parallel branches 512, 513, 514, and 515 are then merged with the fifth branch 516, which corresponds to the input of the 3D spatial pyramid pooling block, through a summation operation 517 to generate the output of the 3D spatial pyramid pooling block. That is, branch 516 skips the operations of the four parallel branches 512, 513, 514, and 515.

[0194] Figure 4a It shows Figure 3a Possible exemplary configurations of the memory blocks 240, 250, 260, and 270 of the decoder 200.

[0195] It may include a memory, denoted as storage pool 4010, in which image features / feature tensors extracted from a predetermined number K of previously captured / pre-processed images can be stored.

[0196] Past image feature storage pool 4010 can, for example, store the features of each of the last K images. Figure 2a The 2D encoder shown extracts features / feature tensors from the captured image, where K is, for example, a natural multiple of the number of focal positions N.

[0197] Alternatively, image features 4000 of the currently processed image at a given scale can be stored in storage pool 4010. These features are three-dimensional tensors of shape (C, H, W), with channel dimension C, height dimension H, and width dimension W.

[0198] The memory block may also include a feature alignment block 4020, which can take as input the features / feature tensors stored in the memory pool 4010 (e.g., features / feature tensors extracted from the K previously captured / previously processed images) along with the features / feature tensors extracted from the currently processed image, and output a four-dimensional tensor 4020 of shape (C, N, H, W) representing the image features of each focal position / each focal plane aligned with the last chronologically ordered focal position (i.e., the focal position of the currently processed image).

[0199] Here, C again refers to the channel dimension, N refers to the focal position dimension, H refers to the height dimension, and W refers to the width dimension of the currently processed image / image feature / image feature tensor / feature map.

[0200] Figure 4b An exemplary overview of the configuration of the aforementioned exemplary feature alignment block 4020 is shown.

[0201] The exemplary feature alignment block 4020 has two inputs: a three-dimensional image feature / three-dimensional feature tensor 4040 from the currently processed image; and a four-dimensional tensor 4050, which represents image features extracted from a predetermined number K previously captured / previously processed images and has been stored in a past image feature storage pool (e.g., in past image feature storage pool 4010).

[0202] The exemplary feature alignment block 4020 also includes at least one feature alignment head 4060 and a feature combination operator 4070, such as a summator, for generating as output a multidimensional tensor representing the image features of all processed images aligned with the image features of the currently processed image, i.e., a four-dimensional tensor 4030, 4080 of shape (C, N, H, W), which represents the image features of each focal position / each focal plane aligned with the last chronologically ordered focal position (i.e., the focal position of the currently processed image).

[0203] The feature alignment head 4060 divides the above input into patches of different resolutions, that is, patches at height h compared to the input features. p and width w p The input features can be patched with different sizes, ranging from a 1x1 patch (meaning the input features remain unchanged) to a HxW patch (meaning the entire input feature tensor will be treated as a patch).

[0204] Figure 4c An exemplary configuration of an exemplary feature alignment header is shown, such as feature alignment header 4060 from feature alignment block 4020, which can be used for Figure 3a The exemplary decoder 200 of the convolutional neural network architecture shown.

[0205] The input to the current image feature / feature tensor 4090, i.e. the input to the image features extracted from the currently processed image, is fed to the (first) four-dimensional coding block 4110 via branch 4091, which, as previously described and further detailed below, embeds temporal, spatial, and focal position information into the image features 4090 extracted from the currently processed image.

[0206] The input of past image features 4100, which are image features extracted from previously captured images, such as image features extracted from a predetermined number of K previously captured / pre-processed images, is fed through branch 4101 to a separate (second) four-dimensional encoding block 4190, which embeds temporal, spatial and focal position information into the features extracted from the previously captured images.

[0207] For example, as mentioned earlier, the four-dimensional code E can be constructed based on the following two equations:

[0208]

[0209]

[0210] Where α is a correction constant, preferably greater than C (number of channels or channel dimension), x and y are spatial pixel coordinates, t is time, i.e., the time position / time point / time stamp / time index of the captured image from which image features are extracted, where t∈[0, K-1], where K represents the number of previously captured images, such as a predetermined number of captured images, d∈[0, N-1] is the focal plane position / focal position / focal position index of the given image to be encoded, and N is the total number of images or focal positions, such as the number of images in the focus plan or the sum of the number of images in the focus plan and the number of images derived from the images in the focus plan, where the derived images can be derived by interpolation or extrapolation based on the images captured according to the focus plan, and i∈[0, C / 2] is the index used to divide the number of channels into even and odd channels for encoding.

[0211] Encoded by example E 2i，x，y E 2i+1，x，y The exemplary encoding E can also take into account a given patch width w p and patch height h p Resolution, i.e.

[0212] The exemplary encoding can be applied to the currently processed image through addition. The image features / feature tensor 4090 are applied, and each image feature / feature tensor 4100 from the previously captured images is applied, that is, each image feature / feature tensor from the past K images is applied. To obtain and as follows.

[0213] The four-dimensional coded block 4110 can be obtained through the following formula.

[0214]

[0215] Furthermore, the four-dimensional coded block 4190 can be obtained using the following formula.

[0216]

[0217] in, This represents the encoding of image features / feature tensors from K previously captured images.

[0218] After the current image features are four-dimensionally encoded by the four-dimensional encoding block 4110, a sequence 4121 of two-dimensional convolutional layers (Conv) with batch normalization (BN) is applied to the EF to obtain the EF along the output branch 4120 of the four-dimensional encoding block 4110. query .

[0219] Similarly, after four-dimensional encoding of past image features through four-dimensional encoding block 4190, a sequence 4131 of two-dimensional convolutional layers (Conv) with batch normalization (BN) is applied to the EPF to obtain the EPF along the output branch 4130 of the four-dimensional encoding block 4190. key .

[0220] Here, the superscripts query and key are used only as exemplary references to the concepts of the retrieval system, as will be explained further below.

[0221] The outputs from the output branches 4120 and 4130 are fed as inputs to the patch-wise similarity block 4150.

[0222] This block 4150 first uses the three-dimensional tensor Reshape into a two-dimensional matrix And the four-dimensional tensor plastic surgery

[0223] Then, calculate the EF of the integer shape. query With EPF key The similarity between each of the K feature tensors. This similarity operation can be computed by a patch-style similarity block 4150, for example, EF query′ =EF′ and EPF key′ =EPF′ is as follows:

[0224]

[0225] in The similarity score is the image feature of the currently processed image compared to the image features of each of the K past / previously captured images.

[0226] Specifically, Sim k，i，i′ This can be understood as describing the similarity between patch i of the feature tensor of the currently processed image and patch j of the feature tensors of K past / previously captured images.

[0227] EF′ and EPF′ can have [(H*W) / (w p *h p ), W p *h p The shape of *C], wp and h p These represent the width and height of the patch, respectively. For example, assuming the patch size is [1, 1], the shape will be [H*W, C]. Therefore, the range of index i and index i′ is [0, (H*W)-1], and the range of index j is [0, C-1].

[0228] Then, the similarity scores are converted into probabilities by the patch-style similarity block 4150:

[0229]

[0230] in, It is a normalized similarity score, which has the following properties:

[0231]

[0232] The normalized similarity score Sim′ represents the output 4151 of the patch similarity block 4150 after processing the input received from the branch 4120 after the first four-dimensional (4D) coding block 4110 and from the (first, upper) branch 4130 after the second four-dimensional (4D) coding block 4190, wherein the first four-dimensional (4D) coding block 4110 processes image features extracted from the currently processed image, and the second four-dimensional (4D) coding block 4190 processes image features extracted and stored from previously captured images, for example, image features extracted and stored from a predetermined number of captured images (e.g., from the past K images).

[0233] For completeness, it should be noted that the similarity scores described herein are merely exemplary, and other similarity functions can also be used to derive a similarity measure between currently processed image features and previously processed and stored image features. Instead of the exemplary Euclidean similarity described above, other similarity functions can be applied, such as cosine similarity or similarity operations using matrix multiplication or any other function capable of comparing two samples.

[0234] Another (second, lower) branch 4140 of the second four-dimensional (4D) coding block 4190 includes a first sequence 4141, which includes a two-dimensional convolutional layer (Conv) and a batch normalization (BN) operation, which gives the output. Then it is shaped using the reshape operation / layer (Reshape) 4142.

[0235] The branch 4140 also includes a matrix multiplication operation / layer 4143 (Matmul), in which the normalized similarity score Sim′ from the patch-like similarity block 4150 and the EPF v′ Multiply to obtain

[0236]

[0237] Then AF′ is further shaped into H and W correspond to the input image 101, which are the height and width dimensions of the image being processed.

[0238] Here, the superscripts v and v′ are only used to distinguish EPFs from branch 4140. v and EPF v With EPF from branch 4130 key and EF from branch 4120 query .

[0239] This shaping can be part of a matrix multiplication operation / layer 4143 (Matmul), or it can be performed in another separate shaping operation / layer (not shown).

[0240] Then, AF is grouped along the first dimension K through block / operation / layer 4160 to group features corresponding to the same focal position, thereby obtaining in,

[0241]

[0242] Then, after the grouping, all information from the extracted features is combined through a reduction and summation operation / layer 4170 (reduction and summation):

[0243]

[0244] by As an example, this is an example of at least one multidimensional tensor representing the image features of all processed images aligned with the image features of the currently processed image (i.e., the image features of the focal positions of all processed images). As previously mentioned, it is also possible to generate a multidimensional tensor that does not represent all the image features of all processed images, but rather represents at least some of the image features of processed images / previously captured / past images aligned with the image features of the currently processed image.

[0245] The memory block and feature alignment head described herein can be understood as forming a data structure model for a retrieval system, wherein image features can be stored in a key-value pair structure that can be queried to align previously processed and stored image features with the image features of the currently processed image.

[0246] For example, the value of the key-value pair structure can be understood as a four-dimensional tensor of the image features of the previously processed and stored image after applying a sequence 4141, including a two-dimensional convolutional layer (Conv) with batch normalization (BN), along the lower branch 4140. The content represented by / is understood as The content represented by / can be understood as the four-dimensional tensor of the image features of the previously processed and stored image after applying the sequence 4131, which includes a two-dimensional convolutional layer (Conv) with batch normalization (BN), along the upper branch 4140 following the 4D positional encoding block 4190. The content represented by / .

[0247] A query can be understood as a three-dimensional tensor. The key, i.e., the EF of the output branch 4120 of the four-dimensional coded block 4110 that processes image features from the currently processed image. query The content represented by / .

[0248] In other words, four-dimensional tensor This represents a set of keys in a retrieval system, relative to a query. It is mapped to retrieve a specific value, content, or key from the set of keys that best matches the query.

[0249] For completeness, it should be noted that the weights of the convolutional layers applied in branches 4130 and 4140 may differ. These weights, for example, can be learned / optimized during the training of the convolutional network.

[0250] Figure 5 An exemplary flowchart of a method 800 for extracting depth information from multiple images captured by a camera at different focus positions is shown, which may include one, some, or all of the following steps.

[0251] • Capture a sequence of images of scene 801 at different focus positions using a camera according to a predetermined focus plan that specifies the time sequence of the camera focus positions. The focus plan may include any combination of multiple unique and / or non-unique (e.g., repeating) focus positions.

[0252] • Image features of a predetermined number of captured images are extracted using a machine learning algorithm including a convolutional neural network, and the extracted image features are stored. For example, the convolutional neural network includes, for example, […]. Figure 2a , 2b The configurations exemplarily described in 2c, 2d, 2e, 3a, 3b, 3c, 3d, 4a, 4b, 4c, and the storage may include storing the features, for example, within memory blocks 240, 250, 260, 270, or in feature storage pool 4010.

[0253] • After extracting and storing image features from the predetermined number of captured images, the captured images whose image features have not yet been extracted are processed by a machine learning algorithm. The captured images represent the images currently being processed, such as input image 101.

[0254] The processing includes extracting image features from the currently processed image using machine learning algorithms and storing the extracted image features.

[0255] The processing also includes aligning image features stored from previously captured images with image features of the currently processed image, wherein, for example, the alignment is performed by a feature alignment header of a memory block, such as... Figure 3a , 4a As exemplarily described in 4b and 4c.

[0256] The processing further includes generating at least one multidimensional tensor (which represents the image features of all processed images aligned with the image features of the currently processed image) as, for example, the tensor described above.

[0257] • Generate a two-dimensional depth map using the focal location specified in the predefined focusing plan and at least one generated multidimensional tensor.

[0258] Figure 6 A schematic example of a possible training protocol for a machine learning algorithm 630 is shown, which includes a convolutional neural network with the architecture described above.

[0259] Here, the training samples include multiple / one sequence 600 of captured images, which are focused at different focal positions according to a focusing scheme 620 from multiple different scenes in the real physical world. The training samples can be processed according to the steps described above to obtain a sequence 640 of focus probability maps, one for each image after processing a predetermined number of captured images.

[0260] The captured images may have been taken with the same camera or different cameras. In other words, the method described here is independent of the type of camera, i.e., it is not limited to using a specific type of camera.

[0261] The scenes captured in the image sequence 600 of the training samples can be static or dynamic, meaning there can be movement between images, such as due to the movement of objects or subjects in the scene and / or due to camera movement, such as vibration caused by the camera being held in the user's hand or by the camera changing its position.

[0262] The obtained focus probability map is remapped to the actual distance using the focus position from the known focusing plan 620.

[0263] The result is a sequence of predicted / estimated depth maps, which are then used together with a sequence of ground true depth maps 610 (i.e., known / expected depth maps) as input to a loss function 660.

[0264] The loss function 660 is a measure of how different the estimated / predicted depth map is relative to the expected true depth map of the known ground.

[0265] Run the training of machine learning algorithm 630, which includes a convolutional neural network, until the loss function has reached the expected / specified minimum and the optimal model parameters of the convolutional neural network have been determined.

[0266] The loss function can be minimized through optimization techniques (e.g., using the gradient descent algorithm).

[0267] However, other optimization techniques, such as simulated annealing, genetic algorithms, or Markov-chain-Monte-Carlo algorithms, can also be applied to minimize the loss function and determine the optimal model parameters for the machine learning algorithm / convolutional neural network from the training.

[0268] To further optimize training, visual cues can be used to better derive semantically correct depth maps. For example, a convolutional neural network can be trained to recognize that when one object occludes another, the occluding object is closer to the camera than the occluded object.

[0269] Next is Figure 1 , 2a 2b, 2c, 2d, 2e, 3a, 3b, 3c, 3d, 4a, 4b, 4c, 5 and 6, wherein the reference numerals denote the following exemplary components and exemplary steps.

[0270] 700 image streams, multiple images

[0271] 710 Focus Plan

[0272] 720 Machine Learning Algorithms Including Convolutional Neural Networks

[0273] 730 Focus Probability Chart

[0274] 740 remapped to actual physical distance

[0275] 750 Estimated / Extracted Depth Map

[0276] A 2D encoder using a 100-convolutional neural network

[0277] 101 Input image / Currently being processed image

[0278] 102 2D Convolutions / Convolutional Blocks / Layers

[0279] 103, 104, 105, 106 Two-dimensional residual convolution / convolutional blocks / layers

[0280] 107 Two-dimensional spatial pyramid pooling blocks / layers

[0281] 108, 109, 110 Two-dimensional multi-scale feature aggregation blocks / layers

[0282] 111, 112, 113, 114: Sequences of blocks / layers in a convolutional neural network

[0283] 111a, 112a, 113a, 114a Convolutional Layers / Blocks / Operations

[0284] 111b, 112b, 113b, 114b Batch Normalization Layer / Block / Operation

[0285] Linear activation function layers / blocks / operations corrected to 111c, 112c, 113c, and 114c

[0286] 115, 116, 117, 118 Output / Extracted Features / Feature Tensors

[0287] 119, 120, 121 Sequences of 2D convolutions / convolutional blocks / layers

[0288] 122 2D Convolution / Convolution Block / Layer Pooling Block / Layer

[0289] 123 Sequence of linear activation function operations for two-dimensional convolutional layers, batch normalization, and correction.

[0290] 124 and 127 two-dimensional convolutional layers and batch-normalized sequences

[0291] 125 Summation Layer / Block / Operation

[0292] 126-corrected linear activation function operation

[0293] 128, 129 2D residual convolution / convolution block / layer branching

[0294] 130 Upsampling Layer / Block / Operation

[0295] Sequence of linear activation function operations for 131 and 133 two-dimensional convolutional layers, batch normalization, and correction.

[0296] 132 Cascaded Layers / Blocks / Operations

[0297] 134, 135, 136, 137, 138 Branches of 2D spatial pyramid pooled blocks / layers

[0298] 139 Summation Layer / Block / Operation

[0299] 200 decoders / decoding layers

[0300] 201, 202, 203, 204 Input features / feature tensors

[0301] Memory blocks 240, 250, 260, 270

[0302] 340, 370, 400 three-dimensional multi-scale feature aggregation blocks / layers

[0303] 320, 350, 380, 410 3D residual convolution / convolution block / layer

[0304] 330, 360, 390, 420 3D spatial pyramid pooled blocks / layers

[0305] Output branches of 3D spatial pyramid pooling blocks / layers: 430, 431, 440, 441, 450, 451, 461

[0306] 280, 290, 300 Intermediate Focus Probability Chart

[0307] 310 Final Focus Probability Chart

[0308] 501, 502 3D residual convolution / convolution block / layer branching

[0309] A sequence of 503 two-dimensional convolutional layers, batch normalization, and corrected linear activation function operations.

[0310] 504 and 505 two-dimensional convolutional layers and batch-normalized sequences

[0311] 506 Summation Layer / Block / Operation

[0312] 507-corrected linear activation function operation

[0313] 508 Upsampling Layer / Block / Operation

[0314] A sequence of 509 and 511 two-dimensional convolutional layers, batch normalization, and corrected linear activation function operations.

[0315] 510 Connection Layer / Block / Operation

[0316] Branches of 3D spatial pyramid pooled blocks / layers: 512, 513, 514, 515, 516

[0317] 518 Summation Layer / Block / Operation

[0318] Image features of 4000 currently processed images

[0319] 4010 Computer-readable storage device for storing image features / features from a previously processed image / feature storage pool.

[0320] 4020 Feature Alignment Block

[0321] 4030 Image features of each focal position / focal plane aligned with the last chronologically ordered focal position (i.e., the current focal position).

[0322] 4040 Image features of the currently processed image

[0323] Image features of 4050 previously processed images

[0324] 4060 Feature Alignment Header

[0325] 4070 Feature Combinator / Block / Layer

[0326] 4080 image features aligned with the last chronologically ordered focal position (i.e., the current focal position) for each focal position / focal plane.

[0327] 4090 Image features of the currently processed image

[0328] Image features of 4100 previously processed images

[0329] 4110, 4190 4D Position Encoding Block / Layer / Operation

[0330] Sequences 4121, 4131, and 4141 with batch normalized 2D convolutional layers

[0331] 4120 Branch of 4D position coding block 4110

[0332] Branches of 4D position coding block 4190, 4130 and 4140

[0333] 4142 Shaping Layer / Block / Operation

[0334] 4143 Matrix Multiplication Layer / Block / Operation

[0335] 4150 Patch-based Similarity Blocks / Layers / Operations

[0336] 4160 is used for grouping focus positions into blocks / layers / operations.

[0337] 4170 Reduction and Summation Operation / Layer

[0338] 4180 Image features of each focal position / focal plane aligned with the last chronologically ordered focal position (i.e., the current focal position).

[0339] 800 Methods for Extracting Depth Information from Multiple Images Taken by a Camera at Different Focal Points

[0340] Methods and steps 801, 802, 803, 804, 805, 806, and 807

[0341] A sequence of images from 600 training samples

[0342] 610 Sequence of known / expected / true ground depth maps

[0343] 620 Focus Plan

[0344] 630 Machine Learning Algorithms Including Convolutional Neural Networks

[0345] The sequence of focused probability maps obtained by 640

[0346] 660 loss function

[0347] 670. Remapping operation to actual physical distance to obtain estimated / predicted depth map.

Claims

1. A computer-implemented method for extracting depth information from multiple images captured by a camera at different focal positions, the method comprising: According to a predetermined focusing plan based on the time sequence of the specified camera's focus positions, the camera captures a sequence of images of the scene at different focus positions. Image features of a predetermined number of captured images are extracted using a machine learning algorithm including a convolutional neural network, and the extracted image features are stored. After extracting and storing image features from the predetermined number of captured images, the machine learning algorithm processes captured images whose image features have not yet been extracted, where the captured images represent the currently processed images. The processing includes extracting image features from the currently processed image using the machine learning algorithm, and storing the extracted image features. The processing further includes aligning image features stored from a previously captured image with image features from the currently processed image. This alignment involves applying four-dimensional encoding to the image features stored from the previously captured image and the image features from the currently processed image. The four-dimensional encoding includes embedding temporal, spatial, and focus location information into the image features from the previously captured image and the image features from the currently processed image. Generate at least one multidimensional tensor, the at least one multidimensional tensor representing some or all of the image features of the processed image aligned with the image features of the currently processed image, and generate a two-dimensional depth map using the focal position specified in the predetermined focusing plan and at least one generated multidimensional tensor, the generation of the two-dimensional depth map comprising: generating at least one multidimensional focus probability map by the machine learning algorithm, and remapping the at least one multidimensional focus probability map to actual physical distance using the focal position specified in the predetermined focusing plan.

2. The method according to claim 1, in, The image features are extracted into a three-dimensional feature tensor comprising width dimension W, height dimension H, and channel dimension C. Wherein, the channel dimension describes the number of feature maps extracted from the image through one or more layers of the convolutional neural network, and The storage of extracted image features includes storing the extracted image features as a list of three-dimensional feature tensors.

3. The method according to claim 1, in, The four-dimensional encoding is non-linear, and The four-dimensional encoding is applied via addition to each of the image features from the currently processed image and the image features stored from the previously captured image.

4. The method according to claim 3, wherein, The four-dimensional encoding is based on the use of trigonometric functions.

5. The method according to claim 1, in, The at least one multidimensional focus probability map is a three-dimensional tensor with a width dimension W, a height dimension H, and a focus position dimension N, wherein the focus position dimension describes the number of focus positions, and Wherein, the size of the width dimension and the height dimension are equal to the size of the width dimension and the height dimension of the input image, wherein the input image is an image of the predetermined number of captured images or the currently processed image.

6. The method according to claim 5, wherein, The remapping of the at least one multidimensional focus probability map to an actual physical distance using the focus position specified in the predetermined focus plan includes: calculating the dot product between each pixel of the at least one multidimensional focus probability map and the focus position in the focus plan.

7. The method according to claim 1, in, The multidimensional tensor generated from at least one of the image features of all processed images aligned with the image features of the currently processed image is a four-dimensional tensor comprising a width dimension W, a height dimension H, a channel dimension C, and a focal position dimension N. The channel dimension describes the number of feature maps extracted from the processed image by one or more layers of the convolutional neural network, and the focal position dimension describes the number of focal positions.

8. The method according to claim 1, in, Extracting image features from the predetermined number of captured images and extracting image features from the currently processed image further includes: extracting image features at different scales using the machine learning algorithm. Wherein, the scale is defined as a fraction of the height of the input image and / or a fraction of the width of the input image, and The input image is either an image of the predetermined number of captured images or the currently processed image.

9. The method according to claim 1, wherein, Image features extracted from the predetermined number of captured images and image features extracted from the currently processed image are stored in a computer-readable memory in a circular buffer capable of holding at least the image features from the predetermined number of captured images.

10. The method according to claim 1, wherein, The predetermined number of captured images is equal to or greater than the number of different focal positions specified by the focusing plan.

11. The method according to claim 1, in, The convolutional neural network is a trained convolutional neural network that has been trained on training samples, which include multiple images focusing on different focal positions for multiple different scenes. Wherein, the scenario is static or dynamic, and Specifically, the parameters of the convolutional neural network are optimized by comparing the estimated depth map generated by the convolutional neural network with the corresponding known ground depth map using a loss function.

12. A computing system, comprising: Computer memory, One or more processors, the computer memory storing instructions that instruct the one or more processors to perform a method according to any one of claims 1 to 11 for extracting depth information from a plurality of images captured by a camera at different focal positions.

13. The computing system according to claim 12, wherein, The computing system is a portable mobile device, including a camera configured to capture images of a scene with different focus positions.

14. A computer-readable storage medium for storing computer-executable instructions, which, when executed by a computer system, perform the method according to any one of claims 1 to 11 for extracting depth information from a plurality of images captured by a camera at different focal positions.

Images