Unsupervised pre-training of geometric vision models

Abstract

To disclose a method for training a machine learning model for a geometric vision task.SOLUTION: A method to be disclosed includes the steps of: executing teacherless prior learning for a model including an encoder and a decoder; and finely adjusting a model initialized by a set of encoder parameters learned in advance for a geometric vision task. The step for executing the teacherless prior learning for the model includes the steps of: acquiring a first image and a second image on different conditions or at different time points; encoding the first image into a first image representation, and encoding the second image into a second image representation by the encoder; converting the first image representation into a converted representation; decoding the converted representation into a reconstructed image by the decoder; and adjusting at least one or more parameters in the encoder and the decoder on the basis of loss minimization. The step of converting the first image representation and the step of decoding the converted representation are based on the first image representation and the second image representation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application claims the benefit of European Patent Application EP22306534.3, filed October 11, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to computer-based image analysis, and more particularly, to systems, methods, and computer-readable media for training machine learning models for geometric vision tasks. [Background technology]

[0003] The description provided herein is intended to generally provide the context of the present disclosure, and the work of the inventors named herein is not admitted, expressly or impliedly, as prior art to the present disclosure, not only to the extent described herein, but also to aspects of the description that did not qualify as prior art at the time of filing the application.

[0004] Human vision is based on two optical sensors (the eyes) and a highly specialized and efficient image analysis engine (the vision-related part of the brain). By analyzing the images detected by the eyes, the brain is able to recognize objects and object edges in the image, estimate object distances, and estimate object speed and future object positions.

[0005] Such image analysis capabilities are crucial for everyday human activities such as driving a car, operating machinery, picking up and throwing objects, hammering a nail, and searching in a crowd.

[0006] Computer-based devices are configured to autonomously perform human activities, and one element of this goal is to provide the computer-based devices with computer vision that mimics human vision.

[0007] Computer vision systems are configured to analyze images in a manner similar to the human visual system. Computer vision systems provide a sufficient approximation to the image analysis capabilities of the human brain to enable computer-based devices to perform advanced vision-related tasks such as object and edge recognition, monocular or binocular depth estimation, optical flow estimation, or pose estimation. Advanced vision-related tasks such as depth estimation, optical flow estimation, pose estimation, etc. are based on an understanding of the three-dimensional (3D) geometry of the depicted scene.

[0008] Hereafter, such advanced vision-related tasks will be referred to as geometric vision tasks. Computer vision-related techniques are used, for example, in autonomous vehicles, autonomous robotic machines, and other devices. Improved computer-based performance for geometric vision tasks will also have a significant impact on other technology fields, such as image-based diagnosis and testing.

[0009] Artificial intelligence (AI)-based image analysis can be used for computer vision. In AI-based image analysis, a trained machine learning model is applied to one or more images to extract analytical data (e.g., depth maps, corresponding pixel pairs, object velocities, etc.) for a specific geometric vision task from one or more images. The quality of the extracted analytical data depends on the training of the machine learning model.

[0010] Typically, a machine learning model includes a number of parameters that can be learned. Training a machine learning model aims to find a set of parameters that optimizes the results of a particular task for a set of training data. The deviation from the optimal result is expressed as a loss (e.g., the value of a loss function). Finding an optimal set of parameters translates to finding a set of parameters that leads to a minimum of the loss function. Since training is based on the deviation of the results achieved for a given task from the optimal result to minimize the loss, the AI ​​system may receive an indication of the optimal result for improvement during the training phase.

[0011] Supervised learning approaches involve annotated training data, including ground truth data that represents the best results for a particular task. Because a significant effort is required to annotate the training data, the amount of annotated data available for a particular task is limited. The problem of limited training data is alleviated in unsupervised training approaches.

[0012] Unsupervised learning can use unannotated training data without explicitly specified ground truth data, and therefore can take advantage of large-scale data storage of unannotated data (e.g., image databases and Internet text storage). However, unsupervised learning may be limited to tasks where the optimal result for a task can be determined without explicit annotation of ground truth data.

[0013] Even if the task on which a machine learning model is trained does not support unsupervised training, improved training results can be obtained by combining unsupervised and supervised learning. Unsupervised pre-training can be used for a variety of advanced tasks in computer vision. In a second training phase, the pre-trained machine learning model (also called the base model) is fine-tuned and trained on a small annotated dataset, which can perform better for some tasks compared to supervised learning approaches and even outperform supervised learning approaches when large annotated datasets are not available.

[0014] Unsupervised learning is suitable for a variety of advanced computer vision tasks, such as image classification and object detection. Such models use additional image data without additional labels. An example of unsupervised learning is contrastive learning, which constitutes a pre-text task by learning a model output that is invariant to data augmentation. Masked Image Modeling (MIM) is an alternative to unsupervised learning, where such models are trained using an auto-complete pre-text task. An encoder encodes a partial view of the image input, obtained by splitting the image into patches and masking some of them, into a latent representation. The unseen patches are then predicted by a decoder based on the latent representation. Such methods are examples of the so-called self-supervised learning, which is a specific approach to unsupervised learning. Tasks where such traditional self-supervised learning excels include single-view semantic segmentation-oriented tasks, such as image classification and semantic segmentation. However, addressing 3D tasks related to the 3D geometry of the depicted scene (e.g., geometric vision tasks) remains challenging. Summary of the Invention [Problem to be solved by the invention]

[0015] To overcome the above-mentioned problems, a computer-implemented method and system, and a computer-readable medium, for training a task-specific machine learning model for a geometric downstream vision task are provided. One approach is based on unsupervised pre-training, specifically tailored by a module for the geometric downstream vision task, with rewards for the use of multi-view geometric cues. The pre-training includes only unannotated image pairs depicting the same scene under different viewpoints and / or different conditions (e.g., lighting, etc.). The pre-text task is specifically tailored such that the pre-text model learns an understanding of the spatial relationship between the two images, and thus the 3D geometry of the depicted scene, in order to correctly perform the pre-text task. The pre-text model includes an encoder and a decoder. The encoder encodes each of the two images of the image pair into a respective (e.g., vector or matrix) image representation in a representation space. Such image representations are also called latent representations, and the respective representation spaces are also called latent spaces. After this, one of the image representations is transformed in the representation space, and the transformed representation is decoded into a reconstructed image.

[0016] As an example of unsupervised pre-training, one image of a pair is partially masked and the masked content is predicted not only from the visible content of the first image but also from the second image. Such an approach is called cross-view completion (CroCo) pre-training. In cross-view completion, ambiguity caused by masking part of one image can be resolved by inferring the masked content from the other unmasked image, which provides some understanding of the spatial relationship of the two images.

[0017] As an example of unsupervised pre-training, the representation of one of the images is transformed in the representation space to align with the representation of the other image. Such an approach is called cross-view alignment (Caiman) pre-training. In cross-view alignment, the transformed representation of a first image is decoded in the reconstructed image as a prediction of a second image. Solving such a task involves some understanding of how the shape of a scene changes between geometric views.

[0018] The presented pre-learning approach does not require any supervised learning, e.g., ground-truth correspondences between training images. The disclosed approach includes adaptations to learn low-level geometric cues from unlabeled data. Thus, the pre-text model can be pre-trained on large unannotated datasets such that the pre-trained pre-text model (base model) learns a general understanding of the 3D geometry of the depicted scene and can therefore be efficiently fine-tuned on a number of geometric vision tasks. [Means for solving the problem]

[0019] In one embodiment, a machine learning method for training a task-specific machine learning model for a geometric downstream vision task is provided, the method includes performing unsupervised pre-training of a pre-text machine learning model including an encoder having a set of encoder parameters and a decoder having a set of decoder parameters, constructing a task-specific machine learning model for the geometric downstream vision task based on the pre-trained pre-text machine learning model including a task-specific encoder having the set of task-specific encoder parameters, initializing the set of task-specific encoder parameters with the set of encoder parameters of the pre-trained pre-text machine learning model, and fine-tuning the task-specific machine learning model initialized with the set of encoder parameters for the geometric downstream vision task. Pre-training the pre-text machine learning model includes obtaining unannotated image pairs including a first image and a second image depicting the same scene but taken in different conditions or different viewpoints, encoding, by an encoder, the first image into a first image representation and encoding the second image into a second image representation, converting the first image representation into a transformed representation, decoding, by a decoder, the transformed representation into a reconstructed image, and tuning the encoder and decoder based on or by tuning a set of encoder parameters and a set of decoder parameters for minimization of a loss function, where converting the first image representation and / or decoding the transformed representation are based on the first image representation and the second image representation.

[0020] In one feature, the unsupervised pre-training is cross-view alignment pre-training of the pre-text machine learning model, and transforming the first image representation includes applying a transformation to the first image representation to determine a transformed representation, the transformation being determined based on the first image representation and the second image representation such that the transformed representation approximates (e.g., is aligned with) the second image representation. The loss function may be based on a metric that quantifies a difference between the reconstructed image and the second image, or a metric that quantifies a difference between the transformed representation and the second image representation.

[0021] In one feature, the unsupervised pre-training is cross-view completion pre-training of a pre-text machine learning model, and performing the cross-view completion pre-training of the pre-text machine learning model further includes dividing the first image into a first disjoint patch set, dividing the second image into a second disjoint patch set, and masking a plurality of patches of the first patch set. Encoding the first image into a first image representation includes encoding, by an encoder, unmasked patches of the first patch set into corresponding respective unmasked patch representations to generate a first patch representation set. Encoding the second image into a second image representation includes encoding, by an encoder, each patch of the second patch set into corresponding respective patch representations to generate a second patch representation set. Decoding the transformed representation into a reconstructed image includes, for each masked patch of the first patch set, generating, by a decoder, a predicted reconstruction for each masked patch based on the first patch representation set and the second patch representation set. The loss function may be based on a metric that quantifies a difference between each masked patch and each predicted reconstruction.

[0022] In one aspect, a machine learning method for generating predicted data for a geometric downstream vision task is provided by fusing cross-view alignment pre-training and cross-view completion pre-training, which may include training a first task specific machine learning model for the geometric downstream vision task using cross-view alignment pre-training of a first pre-text machine learning model, training a second task specific machine learning model for the geometric downstream vision task using cross-view completion pre-training of a second pre-text machine learning model, generating first predicted data for the geometric downstream vision task by applying the trained first task specific machine learning model to at least one image, generating second predicted data for the geometric vision task by applying the trained second task specific machine learning model to the at least one image, determining a first confidence value for the first predicted data and a second confidence value for the second predicted data, and generating resultant predicted data for the geometric vision task by fusing the first predicted data and the second predicted data based on the first confidence value and the second confidence value. The method may include iterating unsupervised pre-training of pre-text machine learning for the first and second pre-text machine learning models prior to fusing the first and second pre-text machine learning models in a task-specific machine learning model, where the iterating unsupervised pre-training includes pre-training the first pre-text machine learning model using cross-view completion pre-training and pre-training the second pre-text machine learning model using cross-view alignment pre-training.

[0023] In one aspect, a method is provided that includes a computer-implemented method for performing a geometric vision task. The method includes receiving one or more input images, processing a first input image through the geometric vision task, and processing a second input image through a neural network trained by any one of the computer-implemented methods described herein. The trained neural network generates an output for the geometric vision task. In one aspect, the geometric vision task is an estimation of relative pose, and the output is a relative motion of the first input image and the second input image (e.g., a translation between the two images - including relative rotation and deformation of the views of the images). In one aspect, the geometric vision task is an estimation of depth, and the output is a relative distance to a camera, and the depth estimation method may determine a depth map depicting the scene from the first input image (monocular view of the scene) or the first input image and the second input image (binocular view of the scene). In a third embodiment, the geometric vision task is an estimation of optical flow, and the output is a motion of identified visual features of the depicted scene between the first input image and the second input image.

[0024] In one aspect, one or more computer-readable media are provided having computer-readable instructions recorded thereon that, when executed by one or more processors, cause the one or more processors to perform the methods described herein.

[0025] In one aspect, a system includes one or more processors and a memory that, when executed by the one or more processors, causes the one or more processors to perform the computer-implemented methods described herein.

[0026] In one aspect, a computer-implemented machine learning method is described for training a task-specific machine learning model for a geometric downstream vision task, the method including: performing unsupervised pre-training of a machine learning model including an encoder having a set of encoder parameters and a decoder having a set of decoder parameters, constructing a task-specific machine learning model for the geometric downstream vision task based on the pre-trained machine learning model, the task-specific machine learning model including a task-specific encoder having the set of task-specific encoder parameters, initializing the set of task-specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model, and fine-tuning the task-specific machine learning model initialized with the set of encoder parameters for the geometric downstream vision task, where performing unsupervised pre-training of the machine learning model includes obtaining unannotated image pairs including a first image and a second image depicting a same scene and taken under different conditions or different viewpoints, encoding, by an encoder, the first image into a first image representation and encoding the second image into a second image representation, converting the first image representation into a transformed representation, decoding, by a decoder, the transformed representation into a reconstructed image, and tuning at least one parameter of the encoder and the decoder based on loss minimization. Here, transforming the first image representation and decoding the transformed representation are based on the first image representation and the second image representation.

[0027] In additional features, the unsupervised pre-training of the machine learning model is cross-view alignment pre-training, and transforming the first image representation includes applying a transformation to the first image representation to generate a transformed representation, the transformation being determined based on the first image representation and the second image representation such that the transformed representation approximates the second image representation.

[0028] In an additional feature, the unsupervised pre-training of the machine learning model is cross-view alignment pre-training, and the loss is based on a metric that quantifies the difference between a reconstructed image and a second image.

[0029] In an additional feature, the unsupervised pre-training of the machine learning model is cross-view alignment pre-training, and the loss is based on a metric that quantifies the difference between the transformed representation and the second image representation.

[0030] In an additional feature, the first image representation is a first set of n vectors:

number

number

[0031] In additional features, the transformation is a D-dimensional rotation and the transformation matrix (Ω) is a D-dimensional rotation matrix, and the transformation matrix (Ω) is established based on aligning equilateral portions of the vectors of the first vector set with equilateral portions of each vector of the second vector set.

[0032] In an additional feature, the method further comprises determining a transformation matrix (Ω) based on the following equation:

[0033]

number

[0034] where x i,i equiv is a vector (x i,j ) is the equilateral part of x 2,i equiv is a vector (x 2,i ) and SO(D) denotes the D-dimensional rotation group.

[0035] In additional features, the unsupervised pre-learning is cross-view complete pre-learning, and performing cross-view complete pre-learning of the machine learning model further comprises: partitioning the first image into a first disjoint patch set; partitioning the second image into a second disjoint patch set; and masking at least one of the patches of the first patch set; encoding the first image into a first image representation comprises encoding, by an encoder, each unmasked patch of the first patch set into a corresponding respective unmasked patch representation to generate a first patch representation set; encoding the second image into a second image representation comprises encoding, by an encoder, each patch of the second patch set into a corresponding respective patch representation to generate a second image representation; and decoding the transformed representation comprises, for each masked patch of the first patch set, generating, by a decoder, a predicted reconstruction of the respective masked patch based on the transformed representation and the second patch representation set; and a loss function is based on a metric that quantifies a difference between each masked patch and the respective predicted reconstruction.

[0036] In an additional feature, converting the first image representation into the transformed representation further includes, for each masked patch of the first patch set, patching the first patch representation set with the respective learned representation of the masked patch.

[0037] In an additional feature, each learned representation includes a set of representation parameters.

[0038] In additional features, generating the predicted reconstruction of the masked patch of the first patch representation set includes decoding, by a decoder, the learned representation of the masked patch into the predicted reconstruction of the masked patch, where the decoder receives the first patch representation set and the second patch representation set as input data and decodes the learned representation of the masked patch based on the input data, the method further including adjusting the learned representation of the masked patch by adjusting a respective set of representation parameters.

[0039] In additional features, adjusting each set of representation parameters includes adjusting the set of representation parameters based on minimization of a loss.

[0040] In additional features, fine-tuning the task-specific machine learning model includes performing supervised fine-tuning of the task-specific machine learning model.

[0041] In an additional feature, fine-tuning the map includes i) applying a task-specific machine learning model to the one or more annotated images as ground truth data for the geometric downstream vision task to generate output data according to the geometric downstream vision task, and ii) tuning the task-specific machine learning model by adjusting a set of task-specific encoder parameters of the task-specific machine learning model based on minimizing a task-specific loss based on a metric that quantifies a difference between the generated output data and the ground truth data.

[0042] In an additional feature, the method further comprises repeating steps i) and ii) a predetermined number of times or until a task-specific loss minimum is reached.

[0043] In an additional feature, the method further includes applying the fine-tuned task-specific machine learning model to the one or more new images to extract predictive data from the one or more new images for a geometric downstream vision task.

[0044] In an additional feature, the geometric downstream vision task is relative pose estimation.

[0045] In additional features, applying the fine-tuned task-specific machine learning model includes applying the fine-tuned task-specific machine learning model to the new image pair to extract relative rotation and relative transformation between images of the new image pair as predicted data, where the images of the new image pair depict two views of a same scene.

[0046] In an additional feature, the geometric downstream vision task is depth estimation.

[0047] In additional features, applying the fine-tuned task-specific machine learning model includes applying the fine-tuned task-specific machine learning model to the one or more new images to extract one or more depth maps from the one or more new images as predicted data.

[0048] In an additional feature, the geometric downstream vision task is the estimation of optical flow.

[0049] In additional features, applying the fine-tuned task-specific machine learning model includes applying the fine-tuned task-specific machine learning model to a new image pair including a new first image and a new second image to generate a plurality of pixel pairs as predicted data, each pixel pair including a pixel of the new first image and a corresponding pixel of the new second image, wherein the new first image and the new second image depict a same scene under different conditions or different perspectives.

[0050] In one aspect, a computer-implemented machine learning method is described for generating predicted data for a geometric vision downstream task, the method including: training a first task specific machine learning model for the geometric vision downstream task using cross-view alignment pre-training of a first machine learning model, training a second task specific machine learning model for the geometric vision downstream task using cross-view completion pre-training of a second pre-text machine learning model, generating first predicted data for the geometric downstream vision task by applying the trained first task specific machine learning model to at least one image, generating second predicted data for the geometric downstream vision task by applying the trained second task specific machine learning model to the at least one image, determining a first confidence value for the first predicted data and a second confidence value for the second predicted data, and generating resultant predicted data for the geometric vision task by fusing the first predicted data and the second predicted data based on the first confidence value and the second confidence value.

[0051] In one feature, a computing system includes one or more processors and a memory including code that, when executed by the one or more processors, performs unsupervised pre-training of a machine learning model including an encoder having a set of encoder parameters and a decoder having a set of decoder parameters; configure a task-specific machine learning model based on the pre-trained machine learning model, the task-specific encoder having a set of task-specific encoder parameters, for a geometric downstream vision task; initialize a set of task-specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model; and, for the geometric downstream vision task, initialize a set of task-specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model. and fine-tuning a task-specific machine learning model initialized with the set, where performing unsupervised pre-training of the machine learning model includes obtaining unannotated image pairs including a first image and a second image depicting a same scene but taken in different conditions or from different viewpoints, encoding, by an encoder, the first image into a first image representation and encoding the second image into a second image representation, converting the first image representation into a transformed representation, decoding, by a decoder, the transformed representation into a reconstructed image, and tuning at least one parameter of the encoder and the decoder based on loss minimization, where transforming the first image representation and decoding the transformed representation are based on the first image representation and the second image representation.

[0052] Further areas of applicability of the present disclosure will become apparent from the specific details, claims, and drawings. The specific details and specific examples are illustrative only and are not intended to limit the scope of the present disclosure. [Brief description of the drawings]

[0053] The present disclosure will become more fully understood from the detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a functional block diagram illustrating an example of a procedure for training a machine learning model for a geometric vision task. [Diagram 2]FIG. 1 is a functional block diagram illustrating an example of a procedure for unsupervised pre-training of a machine learning model for a pre-text task using unannotated image pairs. [Diagram 3] FIG. 1 is a functional block diagram illustrating an example of a procedure for self-supervised cross-view completion pre-training of a pre-text machine learning model using unannotated image pairs. [Figure 4] FIG. 1 is a functional block diagram illustrating an example procedure for self-supervised cross-view alignment pre-training for pre-text machine learning using unannotated image pairs. [Diagram 5] 1 is a flow chart illustrating an example method for training a task-specific machine learning model for a geometric downstream vision task. [Figure 6] 1 is a flow chart illustrating a method for unsupervised pre-training of a pre-text machine learning model for a pre-text task. [Figure 7] 1 is a flow chart illustrating a method for self-supervised cross-view completion pre-training of a pre-text machine learning model. [Figure 8] 1 is a flow chart illustrating a method for self-supervised cross-view completion pre-training of a pre-text machine learning model. [Figure 9] 9 is a flowchart illustrating a method for generating predicted data by a geometric downstream vision task that combines the self-supervised cross-view completion pre-training of FIG. 7 and the self-supervised cross-view alignment pre-training of FIG. 8 . [Figure 10] 1 is a functional block diagram illustrating an example of a system architecture.Reference numbers in the drawings are used throughout to identify similar or identical elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] Described herein are systems, computer readable media, and methods for training task-specific machine learning models for geometric vision tasks. For purposes of explanation, numerous embodiments and specific details are presented to provide a thorough understanding of the embodiments. The embodiments as defined by the claims may include some or all of these features alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein. Exemplary embodiments are described with reference to the drawings, in which like elements and structures are designated with like reference numerals.

[0055] The present invention relates to a method, system, and computer-readable medium for training a machine learning model for a geometric vision task (e.g., a three-dimensional (3D) geometric task in computer vision). The training is performed in two stages: in the first stage, a pre-learning stage, a pre-text machine learning model is trained by an unsupervised training module on the pre-text task using a training dataset that includes a large number of image pairs (e.g., more than 1000 pairs).

[0056] The pre-text task may be designed for unsupervised learning of 3D geometric cues from image pairs, where each image of an image pair depicts the same scene but from different viewpoints or in different conditions (e.g., different focal lengths, different lighting conditions, different seasons). A second task-specific machine learning model is constructed that is specifically built for the geometric downstream vision task. The geometric downstream vision task is an advanced 3D geometric task in computer vision where learning (fine-tuning) involves supervised training using annotated images. Some of the parameters of the task-specific machine learning model are initialized to the values ​​of the corresponding parameters of the pre-trained pre-text machine learning model. In the second training phase, the fine-tuning phase, the task-specific machine learning model is trained by a training module together with supervised training using a training dataset that includes annotated images.

[0057] 1 is a functional block diagram illustrating an example of a two-stage training process performed by a training module 50 for a machine learning model. Training begins with the training module 50 performing unsupervised pre-training 110 of a pre-text learning model 130 on the pre-text task. The pre-text machine learning model 130 includes a number of learnable parameters θ 132. The pre-training process 110 includes the training module 50 iteratively adapting the values ​​of the parameters 132 such that the resulting pre-text machine learning model 130 continually achieves better results on the pre-text task.

[0058] The pre-training 110 begins with an initial set of parameter values ​​for the parameters (θ) 132. The initial set of values ​​for the parameters (θ) 132 may be selected as a common (e.g., predetermined) value for all parameters (θ) 132, the initial parameter values ​​may be set randomly such that the initial set of parameter values ​​includes a random value for each parameter of the multiple parameters (θ) 132, or may be set in any other suitable manner.

[0059] During pre-learning 110, the pre-text machine learning model 130 receives a pair of unannotated images from the training module 50, including a first image (I1) 120 and a second image (I2) 122. The first image 120 and the second image 122 depict at least partially the same visual content (e.g., the same scene, the same object, the same person, or the same architectural structure), but from different perspectives or in different conditions (e.g., lighting, etc.). In this context, the perspective from which a scene is depicted may refer to the position of a camera and / or the angle of a camera relative to the depicted scene at which an image depicting the scene is taken. The pixel sizes of the first image 120 and the second image 122 may be the same or different. The images 120, 122 do not include annotations with ground truth data by the pre-text task.

[0060] Based on the pair of first image 120 and second image 122, the pre-text machine learning model 130 generates a reconstructed image (I R ) 140. Reconstructed image 140 may be a reconstruction of the first image or the second image, or may be a reconstruction of an image derived from the first image and / or the second image by applying a predefined transformation to the first image and / or the second image. Such predefined transformations may include applying one or more color transformations to one or both of the images, such as a grayscale transformation, applying one or more geometric transformations to one or both of the images, determining a combination of the first and second images, such as the difference for each pixel between the images, etc.

[0061] The pre-text machine learning model 130 converts the images 120, 122 of an image pair into a representation of the images in a mathematical representation space (latent space), also called a latent representation. The representation may be, for example, a vector or a matrix.

[0062] According to the particular pre-text task, the pre-text machine learning model 130 performs a transformation on the image representation in the representation space to generate a final representation of the reconstructed image 140. The final representation of the reconstructed image 140 is transformed back to the image space, thereby generating the reconstructed image 140. Based on the reconstructed image 140 and / or the representation of the reconstructed image 140, a pre-text loss 142 is determined by the training module 50, which represents the quality of the result achieved by application of the pre-text machine learning model 130 to the images of the image pair. The loss may be determined, for example, based on a comparison of the result (the reconstructed image 140 and / or the representation of the reconstructed image 140) to an expected result for the pair.

[0063] In general machine learning, the loss refers to the value of a loss function (also called a "cost function"), which is a function that maps the value of an event or one or more variables to a real number that represents some "cost" associated with the event. The goal of the pre-learning phase is for the training module 50 to modify the internal learnable parameters (θ) 132 of the pre-text machine learning model 130 to minimize the pre-text loss 142 (i.e., the value of the loss function). The loss function may be a complex real-valued scalar function of each of the learnable parameters (θ) 132 of the pre-text machine learning model 130. The optimization of the loss function (converged to a minimum value) is performed by the training module 50 by back-propagation 144 of the loss gradient, which is obtained by the training module from the partial derivative of the loss function with respect to the learnable parameters (θ) 132 of the pre-text machine learning module 130. Such loss gradients are backpropagated by the training module 50 to the respective learnable parameters (θ) 132, in that they are used to modify (or adapt or update) the learnable parameters (θ) 132 of the pre-text machine learning model 130 in order to generate lower losses 142 in the next iterative pre-learning phase.

[0064] In the next iteration, the machine learning model 130 with the modified learnable parameters (θ) 132 uses the modified learnable parameters 132 as described above to generate another reconstructed image 140 based on the same or a different image pair. The transformation of this image pair to a representation, the transformation in the representation space, and the generation of the reconstructed image 140 are calculated using the modified learnable parameters (θ) 132. After this, the modified pre-text machine learning model 130 becomes suitable to perform pre-text tasks on the image pair, leading to a lower loss 142 and a smaller adaptation of the learnable parameters (θ) 132 in the next backpropagation of the loss gradient until the loss function converges to a minimum loss 142 and no adaptation of the learnable parameters (θ) 132 is necessary. Once the loss function converges to a minimum, the pre-text machine learning model 130 is successfully pre-trained and the pre-training phase may be completed. In an alternative embodiment, the repeated pre-training is terminated after a predetermined number of iterations rather than being performed until full convergence is reached.

[0065] The second training stage is fine-tuning 150 of the task-specific machine learning model 170 by the training module 50. Here, the term fine-tuning may refer to training performed after the pre-learning stage based on the results of the pre-learning. In various embodiments, the fine-tuning may be performed by the training module 50 in a supervised or partially supervised manner. Here, the term supervised may refer to training based on annotated training data. The annotated training data includes ground truth data that contains additional information regarding the optimal / expected results / outcomes of the specific task for which training is performed. The ground truth data is not used by the machine learning model to be trained 170 but is used by the training module 50 to determine a loss 182 for the specific training task based on the results generated by the machine learning model to be trained 170 given the input.

[0066] Similar to the pre-text machine learning model 130, the task-specific machine learning model 170 includes multiple learnable parameters (θ′) 172. Fine-tuning is performed in the same way as pre-training. However, the task on which the task-specific machine learning model 170 is trained by the training module 50 is a geometric vision task for which more accurate training involves the use of annotated (ground truth) data. This task can be referred to as a downstream task, indicating that the ultimate goal of training is to provide a trained machine learning model optimized to perform this task. The pre-text task is a predecessor task for improving the training of the task-specific machine learning model 170 for the geometric downstream vision task.

[0067] The fine-tuning 150 begins with an initial set of parameter values ​​for the parameters (θ′) 172. The training module 50 initializes the parameters 172 with the initial set. The training module 50 initializes at least a portion of the parameters (θ′) 172 to values ​​of the corresponding portions of the parameters (θ) 132 of the pre-trained, pre-text machine learning model 130. The initial set of values ​​for the remaining portions of the parameters (θ′) 172 that are not initialized to the parameter values ​​of the pre-trained, pre-text machine learning model 130 may be set by the training module 50, for example, to a common (e.g., predetermined) value for all parameters of the remaining portion of the parameters (θ′) 172, may be set randomly such that the initial set of parameter values ​​includes a random value for each parameter of the remaining portion of the parameters (θ′) 172, or may be set by any other suitable method.

[0068] The task-specific machine learning model 170 is applied to one or more annotated images from the training module 50. The example of FIG. 1 shows annotated images (P1, P2) 160, 162. Image (P2) 162 is shown with a dotted line indicating selective characteristics of the second input image. Image (P1) 160 includes ground truth annotations 164 and image (P2) 162 includes ground truth annotations 166. In some embodiments, the two images (P1, P2) 160, 162 may be provided as an image pair, where the ground truth annotations are provided for the pair and not for each image individually. In various embodiments, only a portion of the images are annotated and fine-tuning is performed partially supervised and partially unsupervised. The number of input images varies depending on the downstream task. Applying the task-specific machine learning model 170 to one or more images results in task-specific output data 180. That is, the task-specific machine learning model 170 generates the task-specific output data 180 based on the input images. Based on the task-specific output data 180 and the ground truth data of the one or more annotated input images, a task-specific loss 182 is determined by the training module 50. The task-specific loss 182 represents the quality of the task-specific output data 180 achieved by application of the task-specific machine learning model 170 to the images of the image pair (or input images).

[0069] As in pre-training, the task-specific loss 182 represents the value of a loss function. The goal of the fine-tuning phase is to modify the learnable parameters (θ′) 172 inside the task-specific machine learning model 170 by the training module 50 to minimize the task-specific loss 182 (the value of the loss function). The loss function may be a complex real-valued scalar function of each of the learnable parameters (θ′) 172 of the task-specific machine learning model 170. The optimization of the loss function (converged to a minimum value) is performed by backpropagation 184 of the loss gradient, which is obtained by the training module 50 from the partial derivatives of the loss function with respect to the learnable parameters (θ′) 172 of the task-specific machine learning model 170. Such loss gradients are backpropagated by the training module 50 to each learnable parameter (θ′) 172, in that they are used to modify (or adapt or update) the learnable parameters (θ′) 172 of the task-specific machine learning model 170 to generate a lower loss 182 in the next iterative fine-tuning phase.

[0070] In the next iteration, the task-specific machine learning model 170 with the modified learnable parameters (θ′) 172 is applied to the same or different annotated images to generate different task-specific output data 180. That is, the task-specific machine learning model 170 generates new task-specific output data 180 based on the input image. The modified task-specific machine learning model 170 becomes suitable for performing downstream tasks on one or more images, leading to a lower loss 182 and smaller adaptation of the learnable parameters (θ′) 172 in the next backpropagation of the loss gradient until the loss function converges to a minimum loss 182 and no adaptation of the learnable parameters (θ′) 172 is necessary. Once the loss function has converged, the task-specific machine learning model 170 has been successfully fine-tuned. In various embodiments, the repeated fine-tuning is terminated by the training module 50 after a predetermined number of iterations rather than being performed until convergence is reached.

[0071] Figure 2 shows an example of the pre-learning stage 110 of training in Figure 1. The pre-text machine learning model 130 includes an encoder (module) 230, a representation space block (module) 260, and a decoder (module) 270. The encoder 230 includes a set of learnable encoder parameters, and the decoder 270 includes a set of learnable decoder parameters. Depending on the particular pre-text task, the representation space block 260 may also include a set of learnable parameters. Learnable and trainable may be used interchangeably.

[0072] The pre-text machine learning model 130 is applied to a pair of unannotated images including a first image (I1) 120 and a second image (I2) 122. The first image 120 depicts a visual content at a first time point. The second image 122 depicts at least partially the same visual content (e.g., the same scene, the same object, the same person, or the same architectural structure), but differs from the first image in that the conditions under which the images were generated are different, such as the time at which the same visual content is depicted, the lighting conditions, the season, the depth of field, and the focal length. The pixel sizes of the first image 120 and the second image 122 may be the same or different. The images 120 and 122 do not need to include annotations with ground truth data from a pre-text task, since the pre-learning method is performed unsupervised. In some embodiments, the pre-learning may be performed in a self-supervised manner. Self-supervised pre-learning may refer to a specific example of unsupervised pre-learning, where the loss is determined by the learning module 50 based on unannotated input data (e.g., input images) and does not require ground truth annotations to perform the pre-learning.

[0073] The encoder 230 of the pre-text machine learning model 130 is applied to a first image (I1) 120 and encodes the image into a first representation (R1) 240. The encoder 230 of the pre-text machine learning model 130 is additionally applied to a second image (I2) 122 and independently encodes the second image 122 into a second representation (R2) 250. The representation of an image is an element of a mathematical space (e.g., a vector space). The representation may be a vector, a set of vectors, a matrix, or a tensor.

[0074] The representation space block 260 receives two image representations (R1, R2) 240, 250 and transforms one of the representations (e.g., the representation (R1) 240 of the first image (I1) 120) into a transformed representation 264. As an example, the first representation (R1) 240 is transformed according to the second representation (R2) 250 such that the transformed representation 264 and the second representation (R2) 250 are aligned. As another example, the first representation is transformed by adding a learnable element to the first representation. The transformation of a representation may be dependent on the other representation or independent of the other representation.

[0075] The decoder 270 decodes the transformed representation 264 into a reconstructed image 140. The reconstructed image 140 may be a reconstruction of the first image or the second image, or a reconstruction of an image derivable from the first image and / or the second image. Optionally, the decoder 270 receives an image representation that has not been transformed into the transformed representation 264 as an additional input for decoding the transformed representation 264 into the reconstructed image 140. As an example, if the first image representation (R1) 240 has been transformed into the transformed representation 264, the decoder 270 may receive the second representation (R2) 250 as an additional input for decoding the transformed representation 264 into the reconstructed image 140. In any case, the reconstructed image 140 may be generated based on two image representations (R1, R2) 240, 250 in that the transformation of the first representation (R1) 240 depends on the second representation (R2) 250 and / or the decoding of the transformed representation 264 (as it results from the transformation of the first representation (R1) 240) is based on the second representation (R2) 250.

[0076] After this, a pre-text loss 142 is determined by the training module 50 based on the reconstructed image 140, and a backpropagation 144 is performed by the training module to modify the learnable parameters of the pre-text machine learning model 130 based on minimizing the pre-text loss. This includes, among other things, the training module 50 modifying the learnable parameters of the encoder 230 and the decoder 270. If the representation space block 260 includes learnable parameters, such parameters may also be updated by the training module during the backpropagation 144. The determination of the pre-text loss 142 may be based on the reconstructed image 140, and in particular, may be based on a metric that quantifies the deviation of the reconstructed image 140 from the image that needs to be reconstructed. As an example, the reconstructed image 140 may be a reconstruction of the first image (I1) 120, and in this example, the pre-text loss 142 may be determined by the training module 50 based on a metric that quantifies the deviation of the reconstructed image 140 from the first image (I1) 120. As another example, the reconstructed image 140 may be a reconstruction of the second image (I2) 122, and the pre-text loss 142 may be determined by the training module 50 based on a metric that quantifies the deviation of the reconstructed image 140 from the second image (I2) 122. Additionally or alternatively, the pre-text loss 142 may be determined by the training module 50 based on a metric that quantifies the difference between the transformed representation 264 and the image representations (R1, R2) 240, 250 that have not been transformed to the transformed representation 264.

[0077] Two specific pre-text tasks for unsupervised pre-training 110 of a pre-text machine learning model 130 are described below with reference to FIGS.

[0078] 3 is an exemplary diagram showing a pre-training system for a pre-text task of cross-view completion. The term cross-view completion pre-training may refer to pre-training a pre-text machine learning model for a cross-view completion pre-text task. Here, the abbreviation CroCo is used to mean cross-view completion.

[0079] 4 is an exemplary diagram showing an unsupervised pre-training system for the pre-text task of cross-view alignment. The term cross-view alignment pre-training may refer to pre-training of a pre-text machine learning model for the cross-view alignment pre-text task. Here, the abbreviation Caiman is used to mean cross-view alignment.

[0080] The cross-view completion pre-learning system shown in Fig. 3 is an example of unsupervised pre-learning 110. Hereinafter, the pre-text machine learning model for CroCo pre-learning will be referred to as the CroCo machine learning model. The CroCo machine learning model corresponds to the pre-text machine learning model 130 as shown in Fig. 2. The CroCo machine learning model includes an encoder (module) 330 (corresponding to the encoder 230), a decoder (module) 370 (corresponding to the decoder 270), and a cross-view completion block (module) 360 (corresponding to the representation space block 260).

[0081] The CroCo machine learning model is applied to a pair of unannotated images including a first image (I1) 310 and a second image (I2) 320. The two images depict the same visual content (e.g., the same scene, the same object, the same person, or the same architectural structure) from different perspectives or in different conditions (e.g., lighting conditions, seasonal conditions, depth of field, focal length, etc.). The pixel size of the two images may be the same or different. Because the CroCo pre-training method is performed unsupervised, the images 310, 320 do not need to contain annotations with ground truth data from the CroCo operation.

[0082] The segmentation module 300 segments a first image (I1) 310 into N1 disjoint image patches (p i 1) Set of 312 (P1):

number

[0083] Similar to the first image (I1) 310, the segmentation module 300 segments the second image (I2) 320 into N2 disjoint image patches (p i 2) Set of 322 (P2):

number

[0084] Some of the patches 312 in the set (p1) are masked by the masking module 305, and the set (p1) is divided into a set of masked patches 314 (i.e., patches a, c, d, h, i, j, and k, identified by hatching):

number

number

[0085] Encoder (ε θ ) 330 includes a set of learnable parameters θ. The encoder 330 is applied to the second image (I2) 320 by applying the encoder 330 to the set of patches (p2), thereby converting the set of patches (p2) into a representation (ε θ The set of patches (p2) is encoded into a set of patch representations 352, and a representation (ε θThe encoder 330 generates one patch representation 352 (e.g., feature vector) for each patch 322 in the set of patches (p2) by encoding each patch 322 in the set of patches (p2) individually into a corresponding patch representation 352.

[0086] The same encoder (ε θ ) 330 is also applied to image (I1) 310. However, in this case, the encoder 330 also applies the remaining set of unmasked patches ( ~ p1), which results in a set of unmasked patches ( ~ p1) as the representation (ε θ ( ~ p1)) Encode to 340. Set of patches ( ~ p1) is encoded into a set of patch representations 342, and a representation (ε θ ( ~ p1)) 340. The encoder 330 forms a set of patches ( ~ By separately encoding each patch of p1 into a corresponding patch representation 342, the encoder obtains a set of unmasked patches ( ~ p1), generate one patch representation 342 (e.g., feature vector) for the patch in

[0087] As an example, the encoder 330 may be implemented with a Vision Transformer (ViT) architecture as described in Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale, ICLR, 2021, incorporated herein in its entirety. Image patches may be used as tokens in the ViT backbone. Following the ViT approach, the encoder 330 may generate a set of input tokens (patch 322 or set (p2)) to which a sinusoidal position invading is added. ~ p1) onto the unmasked patch 312, followed by a series of transformer blocks (e.g., Multi-Layer Perceptron (MLP). Self-attention) sinusoidal position inbedding is described in Vaswani et al., Attention is all you need, NeurIPS, 2017, incorporated in its entirety hereby.

[0088] The cross-view completion block 360 is θ ( ~ p1)) 340, and selectively express (ε θ The CroCo completion block 360 receives the representation (ε θ (p2)) 350 unchanged and transmits the representation to the decoder 370. Additionally, the CroCo block 360 θ ( ~ p1) 340 with learned patch representations 362 (e.g., learned feature vectors) corresponding to the masked patches of image (I1) 310, θ ( ~ p1)) 340 is transformed into the expression (ε θ ( ~ p1)')364. As an example, the expression (ε θ ( ~p1) 340 is patched into one learned representation 362 (e.g., learned feature vector) for each masked patch of image (I1) 310. Since the dimensionality of the patch representations used as input by the decoder 370 may differ from the dimensionality of the patch representations generated by the encoder 330, the CroCo block 360 applies a fully connected layer to the patch representations and projects them to a predetermined input dimension for the decoder 370, thereby θ (p2) and ε θ ( ~ Additional transformations may be performed on p1'.

[0089] The cross-view completion block 360 converts the transformed representation (ε θ ( ~ p1)′) 364 to the decoder 370. Additionally, the decoder 370 receives representation (ε θ (p2)) 350 is received. From now on, the decoder will φ This is displayed as a decoder (D φ ) 370 contains a set of learnable parameters (φ). The decoder 370 θ (p2) The transformed expression (ε θ ( ~ Reconstructing p1)')364 into a set of patches (p1):

number

[0090]

number

[0091] As an example, the transformed representation (ε θ ( ~p1)′) 364 is fed to a series of decode transformer blocks (decoder transformer modules (modules with a transformer architecture)). Each block receives (a) an image representation (ε θ ( ~ p1) 340 and an additional learned patch representation 362 corresponding to the masked patch 314 of ̂p1. θ ( ~ Self-attention on the patch representation of p1)'364, (b) ε θ (p2) cross-attention using 340 masked patch representations, and (c) a multi-layer perceptron (MLP).

[0092] A set of learnable encoder parameters (θ) and a set of learnable decoder parameters (φ), and a transformed representation (ε θ ( ~ The parameters of the learned patch representation of p1)') are updated by the training module 50 by backpropagation 144 based on the pre-text loss 142, as described with reference to FIG.

[0093] As an example, the loss of each pretext is i 1 and - The corresponding masked patch (p) of p1 i 1 may be determined by the training module 50 based on a comparison of each patch of ̂p1 and p1 to the unmasked patch based on a metric that quantifies the difference between ̂p1 and ̂p2. As an example, the pre-text loss for images (I1 and I2) may be determined by the training module 50 based on a comparison of each patch of ̂p1 and p1 to the unmasked patch based on a metric that quantifies the difference between ̂p1 and ̂p2, as shown in the following equation: i 1) Pixels and - Averaged over all unmasked patches of p1 - The corresponding masked patch of p1 (p i 1) is evaluated as the mean squared error (MSE) loss with respect to the corresponding pixels of

[0094]

number

[0095] Alternatively, the pretext loss can be calculated for a reconstructed patch ̂p1 according to, for example, the mean and standard deviation of all pixels in a given patch. i 1 and each patch - The corresponding masked patch p of p1 i 1. Based on the normalization value of each patch, the reconstructed patch ̂p i 1 and - Each corresponding unmasked patch p of p1 i The mean squared error (MSE) loss between 1 and - It is determined and averaged by the training module 50 for all unmasked patches of p1.

[0096] The pre-trained CroCo machine learning model may be used to supervisedly fine-tune a task-specific machine learning model for a geometric downstream vision task, as detailed in the fine-tuning process 150 of FIG. 1. Fine-tuning for monocular tasks (e.g., monocular depth estimation, where a depth map is predicted for a single image), which generates predictions for a single image, may be distinguished from fine-tuning for tasks that require multiple images (e.g., two images) as input. In a monocular fine-tuning scenario, an input image may be split into disjoint patches (e.g., 4×4, 8×8, 16×16, or 32×32 pixel patches) by the splitting module 300 and input to the ViT encoder, which is initialized from the parameters of the encoder 330 pre-trained in a self-supervised manner by the training module 50. The ViT encoder is then fine-tuned by the training module 50 for the downstream task. For precise prediction tasks, the task-specific machine learning model may process each patch representation generated by the ViT encoder independently, and a final fully connected layer may be provided as a prediction head that outputs the required predictions for the pixels of each patch representation. Unlike the ViT encoder, this final output layer is trained by the training module 50 without prior preparation.

[0097] CroCo pre-training may be used for downstream tasks (e.g., optical flow) that involve pairs of images as input. In this case, each input image is divided into disjoint patches (e.g., 4x4, 8x8, 16x16, or 32x32 pixel patches) by the division module 300. The task-specific machine learning model includes a task-specific encoder having the same structure as the encoder 330. The task-specific encoder is initialized by the training module 50 with the parameter values ​​of the pre-trained encoder 330. The encoder 330 of the task-specific machine learning model encodes the patches of each image of the image pair into respective representations, each of which includes a set of patch representations.

[0098] The task-specific machine learning model further includes a task-specific decoder having the same structure as the decoder 370. The task-specific decoder is initialized by the training module 50 with the pre-trained parameter values ​​of the decoder 370. The patch representations of the representation of the first image are processed by the decoder 370 using cross-attention computed on the patch representations of the representation of the second image.

[0099] The final layer may be included as a prediction head to project the output of the decoder 370 into a predetermined size and shape.

[0100] For training, the CroCo machine learning model may be pre-trained for 200 or any other suitable number of epochs, such as using the AdamW optimizer. -4 A cosine learning rate schedule with a base learning rate of 0 may be used with an effective batch size of 256. A linear warm-up may be performed for the first 40 epochs. The ViT backbone may be used as an encoder with a set of 12 transformer blocks (modules) with 768 dimensions and 12 heads for self-attention, with patches of 16x16 pixel size. For the decoder, 8 decoder blocks (modules) with 512 dimensions and 16 heads for self- and cross-attention may be used.

[0101] The CroCo machine learning model may be pre-trained on a dataset including image pairs of indoor scenes. The pre-trained encoder may be maintained as an initial encoder of a task-specific machine learning model as part of fine-tuning the pre-trained encoder to perform monocular depth estimation. Both the pre-trained encoder and the pre-trained decoder to estimate optical flow for a downstream task may be maintained as an initial encoder and decoder of a task-specific machine learning model to estimate optical flow. Both the encoder and the decoder are subsequently fine-tuned by the training module 50 to perform optical flow estimation for a given image pair.

[0102] The performance of the pre-trained CroCo model may be evaluated when fine-tuned for a monocular depth estimation task. In such an example, the task-specific machine learning model generates 256 values ​​(depth predictions) per patch. The final output is exponentiated by the model to enforce the depth predictions to be positive. The training module 50 then selectively trains the model based on minimizing the mean squared error (MSE) loss of the predicted values ​​and the ground truth depth values. The pre-trained CroCo model may also be fine-tuned and evaluated to estimate optical flow by predicting two values ​​for the pixels of each patch using a prediction head. The MSE loss may be minimized by the training module 50 during fine-tuning. Experimental results show that the CroCo pre-text task can pre-train task-specific models for geometric downstream vision tasks more effectively than other methods by adopting the advantages of low-level geometric cues.

[0103] [Table 1]

[0104] Experiments were performed on synthetic images of 3D indoor scenes. In each 3D scene, up to 1000 pairs of camera viewpoints with joint visibility greater than 50% were randomly sampled. Such viewpoint pairs may be rendered using the Habitat simulator. As an example, a total of 1,821,391 point pairs may be generated from different indoor scenes for pre-training.

[0105] Also, in one example, 200,000 training (and respective test) sets for downstream tasks (20,000 pairs each) may be generated for different indoor scenes. For monocular depth estimation, downstream task specific models may be fine-tuned on 20,000 images, while 2,000 images may be used for fine-tuning for optical flow estimation. Images of 256x256 pixel size may be generated and a crop of 224x224 size may be used. Acc@1.25 has been reported for monocular depth estimation, subject to the conditions:

number

[0106] In Table 1 above, we described the impact of masking ratio (r). 75% masking provides accuracy in the context of MAE auto-completion. Overall, higher masking ratios such as 90% or 95% improve the performance of the CroCo model.

[0107] In Table 2 below, we additionally show the performance of the CroCo approach when normalizing to the target and the decoder depth, i.e., D φ It is evaluated when varying the number of decoder blocks (D) in . Target normalization can improve the performance of two downstream tasks. While decoder depth can have minimal impact on monocular tasks, a decoder with sufficient depth can be used for tasks that take pairs of images as input, e.g. optical flow, because it also increases the model capacity.

[0108] [Table 2]

[0109] The CroCo pre-training method evaluated for downstream tasks is compared to untrained training and other models (MAE) in Table 3 below. Other models may be pre-trained with supervision for image classification on this dataset and additionally fine-tuned, or pre-trained on the same images and the same dataset used for pre-training by the CroCo approach. For example, the benefit of pre-training on two tasks was observed when compared to untrained training with a difference of about 20% for depth accuracy at a threshold of 1.25. CroCo pre-training provided a gain of about 1% for depth accuracy at 1.25 and outperformed other models on geometric tasks. Additionally, a two-order of magnitude improvement was obtained for optical flow estimation.

[0110] [Table 3]

[0111] The cross-view alignment (Caiman) pre-learning system shown in FIG. 4 is another embodiment of the unsupervised pre-learning 110. The pre-text machine learning model for Caiman pre-learning is also called the Caiman machine learning model. The Caiman machine learning model corresponds to the pre-text machine learning model 130 as shown in FIG. 2. The Caiman machine learning model includes an encoder (module) 430 (corresponding to the encoder 230), a decoder (module) 470 (corresponding to the decoder 270), and a cross-view alignment block (module) 460 (corresponding to the representation space block 260). The encoder 430 includes a set of learnable encoder parameters (θ), and the decoder 470 includes a set of learnable decoder parameters (φ).

[0112] The Caiman machine learning model calculates the source image (I S )410 and the target image (I T) 420. The two images depict the same visual content (e.g., the same scene, the same object, the same person, or the same architectural structure) from different perspectives or under different conditions (e.g., lighting conditions, seasonal conditions, depth of field, focal length, etc.). The pixel sizes of the two images may be the same or different. Because Caiman pre-training is performed unsupervised, the images 410, 420 do not need to contain additional annotations with ground truth data from the Caiman task.

[0113] Source image (I S ) 410 is converted by the encoder 430 into a source representation (x S ) 440. The encoder 430 encodes the target image (I T )420 to the target expression (x T ) 450. In one embodiment, the source representation (x S ) 440 is an ordered set of n K-dimensional vectors 442:

number

number

[0114] As one example of the encoder 430, the source and target representations may be generated using a Perceiver IO-based architecture as described in Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR, 2022, the entirety of which is incorporated herein. For example, a two-dimensional (2D) convolutional layer (e.g., a 5×5, 6×6, 7×7, or 9×9 2D convolutional layer) and a Rectifier Linear Unit (ReLU) are used to generate feature maps that activate the input image (e.g., the source image (I S )410 or target image (I T ) 420), where the feature map may include one feature vector for each pixel of the input image. The dimension of the feature vector may be a predefined parameter of the encoder and may be any suitable number. As an example, the dimension of the feature vector is 32, 64, 128, or 256. The feature map may be divided by the division module 300 into patches of feature vectors (e.g., disjoint quadratic patches of size 8x8, 12x12, 16x16, 20x20, or 32x32). Each patch may be associated with a Fourier position encoding. The patches are then divided into patches of the input image (e.g., source image (I S )410 or target image (I T ) 420 is used to modify a set of n randomly initialized vectors of dimension (K) by cross-attention to generate a representation (x) of the vector, which is updated by applying a block of self-attention layers (e.g., 4, 6, 8, 10, or 12 self-attention layers).

[0115] Source expression (x S )440 and the target expression (x T ) 450 is provided to a cross-view alignment block 460, which aligns the source representation (x S ) 440 to obtain the source representation (xS )Convert 440 to the converted source representation (^x s )to 468. The conversion (h) 466 results in the converted source representation (^x s )468 being the target representation (x T )The set of conversion parameters (Ω) 464 is configured to convert the source representation (xs) 440 such that the converted source representation (^x s )468 approximates (e.g., is aligned with) the target representation (x T )450. To achieve the alignment of the converted source representation (^x S )468 and the target representation (x T )450, the parameter module (g) 462 may determine the parameters (Ω) 464 of the conversion (h) based on the source representation (x s , x T ), or may determine the parameters (Ω) 464 of the conversion (h) in mathematical terms (Ω = g(x

[0116] As an example of the cross-view alignment block 460, each vector:

Number

Number

Number

[0117]

number

[0118] Let SO(D) denote the D-dimensional special orthogonal group containing all possible D-dimensional rotation matrices. The rotation may be estimated using a closed differentiable form as described in Schonemann, A generalized solution of the orthogonal Procrustes problem, Psychometrika, 1966, or Umeyama, Least-squares estimation of transformation parameters between two point patterns, TPAMI, 1991, or Bregier, Deep regression on manifolds: a 3D rotation case study, 3DV, 2021, which are included here in their entirety. This can be done by using a closed differentiable form for the source representation (x S )440 to the target expression (x T ) 450. Specifically, the transformed source representation (^x s) 468 may be determined by the cross-view alignment block 460 as follows:

number

[0119] To solve the cross-view alignment task, the Caiman machine learning model extracts some transformation parameters (rotation matrix (Ω)) that indicate how the source and target images are related. This allows the model to encode changes such as perspective and lighting between views. In one embodiment, the transformation (h) is constrained to have a predetermined number of degrees of freedom. This predetermined number is an adaptive parameter of the Caiman pre-training procedure and may be set to any suitable number. In the exemplary embodiment described above, this limited number of degrees of freedom corresponds to restricting D to a predetermined number (e.g., 24).

[0120] Source expression (x S ) 440 is the converted source representation (^x s ) 468, the decoder 470 converts the converted source representation (^x s )468 Reconstructed Image (I R ) 480. As an example of a decoder, s ) 468 is decoded into a feature map using a patch-based approach. The feature map may contain one feature vector of a predefined dimension (e.g., 32, 64, 128, or 256) for each pixel of the reconstructed image. For each patch of a predefined patch size (e.g., 8x8, 12x12, 16x16, 32x32 pixels) in the feature map, the flattened representation of the feature map value for this patch is sThe flattened representation is generated by performing cross-attention between the Fourier coding of the 2D positions of the patches and the flattened representation (e.g., by a flattening module). The flattened representation is merged (e.g., by a merging module) into a single feature map, and a 2D convolutional layer with a predefined convolution kernel size (e.g., 5 × 5, 7 × 7, 9 × 9) is applied to generate the final reconstructed RGB image. The reconstructed image is then used as a 2D filter to generate the target image (I T ) 420 has the same pixel size.

[0121] The set of learnable encoder parameters (θ) and the set of learnable decoder parameters (φ) are updated by the training module 50 by backpropagation (e.g., backpropagation 144) based on a pretext loss (e.g., pretext loss 142). The loss for Caiman pretraining (Caiman loss) is the linear regression function of the reconstructed image (I R ) 480 target images (I T ) 420, for example, a reconstructed image (I R ) 480 target images (I T ) 420. Additionally or alternatively, the Caiman loss may be determined by training module 50 based on a metric (e.g., a mean squared error metric) that quantifies the deviation of the transformed source representation (̂x s ) 468 target expressions (x T ) 450, for example, the transformed source representation (^x s ) 468 target expressions (x T ) 450. In one embodiment, the Caiman loss may be determined by training module 50 using the following formula:

[0122]

number

[0123] Here, I R (i) is the reconstructed image (I R ) corresponds to a red-green-blue (RGB) value of 480, and similarly, I T (i) is the target image (I T ) corresponds to an RGB value of 420. δ is the Huber loss.

[0124]

number

[0125] An example is δ=0.1, but other suitable values ​​for δ=0.1 can also be used.

[0126] The pre-trained Caiman machine learning model (specifically, the pre-trained encoder 430 and the pre-trained decoder 470) may be used for supervised fine-tuning of a task-specific machine learning model for a geometric downstream vision task, as detailed in the fine-tuning process 150 of FIG. 1. A distinction may be made between fine-tuning for monocular tasks that generate predictions for a single image (e.g., monocular depth estimation, where a depth map is predicted for a single image) and fine-tuning for tasks that use multiple images (e.g., two images) as input. In a monocular fine-tuning scenario, the task-specific machine learning model may include an encoder and a decoder, both of which have the same structure as their counterparts in the Caiman machine learning model. The encoder parameters of the encoder of the task-specific machine learning model are initialized by the parameters of the pre-trained encoder of the Caiman machine learning model. Similarly, the decoder parameters of the decoder of the task-specific machine learning model are initialized by the parameters of the pre-trained decoder of the Caiman machine learning model. A task-specific model for a monocular geometric downstream vision task (e.g., monocular depth estimation) may include a final convolutional layer that outputs a single channel (e.g., depth). This final layer is trained by the training module 50 without prior preparation, whereas the encoder and decoder are initialized by the training module 50 with pre-trained (predetermined) parameters. The encoder is applied to an input image to generate a representation of the input image. The decoder, together with the final layer, generates output data (e.g., depth data).

[0127] CroCo pre-training may be used for downstream tasks that require pairs of images as input, such as relative pose estimation. As an example of relative camera pose estimation, a geometric downstream vision task involves estimating the relative camera displacement between two views of the same scene. Cross-view alignment provides a pre-training task that is close to the problem of relative pose estimation, where the model pre-trained by Caiman can estimate the relative rotation (R∈SO(3)) and relative deformation of two views of a scene given a particular camera:

number

[0128] In a task-specific machine learning model for pose estimation, the two images are independently fed to a task-specific encoder having the same structure as the encoder 430 of the Caiman machine learning model, and initialized by the training module 50 with the parameters of the pre-trained Caiman encoder 430. The two generated representations are input to a parameter module (g) 462 to estimate an alignment parameter (Ω) between the two image representations. This parameter is given as input to a rotation and deformation head, which determines, for example, a 3×3 matrix and a 3D deformation vector (t), respectively. Such a head may be realized using a multi-layer perceptron (MLP) in various embodiments with a predefined number of hidden layers (e.g., 64, 128, 196, 256). The 3×3 matrix is ​​additionally orthonormalized to the rotation matrix (R) using a special orthogonal Procrustes orthonormalization, as described in Bregier, Deep regression on manifolds: a 3D rotation case study”, 3DV, 2021, the entire contents of which are incorporated herein.

[0129] The downstream model is fine-tuned by a training module 50 in a supervised manner to obtain the ground truth relative pose:

number

[0130]

number

[0131] According to one embodiment, λ=0.1 m -2 However, other suitable values ​​may be used.

[0132] The following table shows the experimental results for monocular depth estimation and pose estimation pre-trained with the self-supervised Caiman process described above. The experimental results were achieved by implementing the Caiman model in PyTorch. The Caiman machine learning model may be pre-trained with the Radam optimizer.

[0133] [Table 4]

[0134] Experiments were performed on synthetic images of 3D indoor scenes. In each scene, up to 1000 camera viewpoints with common visibility of 50% or more were randomly sampled. Such viewpoint pairs were rendered using a simulator. Images of size 256x256 were used, and random crops of size 224x224 were used as input images. Overall, 800,000 pairs from different indoor scenes could be used to pre-train the Caiman machine learning model. An example of a few pairs, viewpoints, sampling, and size is provided, but the present application is open to other values.

[0135] For the relative pose estimation downstream task, the training (respective test) set may contain 200,000 pairs (20,000 each) additionally generated in a similar manner from indoor scenes unseen during the Caiman pre-training process. Subsets of different sizes may be considered to evaluate the relative pose estimation performance of the model, and the task-specific machine learning model may be fine-tuned for 100 epochs.

[0136] Table 4 compares the results obtained when training a task-specific machine learning model (including an encoder and decoder) for relative pose estimation without any prior preparation, and when the task-specific model is fine-tuned from a pre-text machine learning model that was pre-trained using the described cross-view alignment process. Table 4 shows that pre-training has a sustained benefit to the model's performance on this task.

[0137] For monocular depth estimation, each task-specific machine learning model may be fine-tuned on 20,000 randomly selected images from the downstream task training set used for relative pose estimation. The fine-tuned task-specific machine learning models may be evaluated on the corresponding test set. Acc@1.25 may be used to evaluate the quality of the predicted depth, which is max(d / - d, - d / d) < 1.25, and - d are the predicted depth and ground truth depth, respectively.

[0138] The performance of Caiman pre-training may be empirically evaluated for the geometric downstream vision task of monocular depth estimation. The task-specific machine learning model outputs a scalar value that is exponentiated (e.g., fed into an exponential function) to force positive depth predictions. The task-specific model may be fine-tuned by the training module 50 to minimize the MSE loss of the depth predictions and the ground truth depth values, e.g., in log space. Table 5 compares the performance obtained when training a task-specific machine learning model (both the encoder and decoder) for monocular depth estimation without pre-training, with the performance of the task-specific machine learning model when Caiman is pre-trained. A performance gain was observed when using the pre-trained model, with the performance increasing from 0.39 to 0.48.

[0139] [Table 5]

[0140] 5 is a flow chart illustrating a computer-implemented method 500 for training a task-specific machine learning model for a geometric downstream vision task. The method begins at step 510 with a training module 50 performing unsupervised pre-learning of a pre-text machine learning model (e.g., the pre-text machine learning model 130). Depending on the embodiment, the unsupervised pre-learning may be self-supervised pre-learning. The pre-text machine learning model includes an encoder (e.g., the encoder 230, 330, or 430) and a decoder (e.g., the decoder 270, 370, or 470). The encoder has a set of encoder parameters, and the decoder has a set of decoder parameters. The unsupervised pre-learning may be performed by the unsupervised pre-learning 110 of FIG. 1.

[0141] At stage 520, a task-specific machine learning model (e.g., task-specific machine learning model 170) is constructed or obtained for a geometric downstream vision task (e.g., a 3D geometric task in computer vision such as depth estimation, optical flow estimation, or relative pose estimation) based on the pre-trained pre-text machine learning model. The task-specific machine learning model includes a task-specific encoder having a set of task-specific encoder parameters. The task-specific machine learning model may further include a task-specific decoder having a set of task-specific decoder parameters. In one embodiment, the task-specific encoder has the same structure as the encoder of the pre-text machine learning model. Furthermore, the task-specific decoder may have the same structure as the decoder of the pre-text machine learning model.

[0142] At step 530, a set of task-specific encoder parameters is initialized with the set of encoder parameters of the pre-trained, pre-text machine learning model. In one embodiment, a set of task-specific decoder parameters is initialized by training module 50 with the set of decoder parameters of the pre-trained, pre-text machine learning model.

[0143] At stage 540, the task-specific machine learning model is fine-tuned and trained by training module 50 for the geometric downstream vision task. In one embodiment, the fine-tuning is performed by supervised fine-tuning based on a training dataset of annotated images or pairs of annotated images, according to supervised fine-tuning 150 of FIG.

[0144] Fine-tuning a task-specific machine learning model for a geometric downstream vision task starts with applying the task-specific machine learning model to one or more images (e.g., annotated images, etc.). For monocular downstream geometric vision tasks (e.g., monocular depth estimation, etc.), the task-specific machine learning model is applied to one input image, and for binocular downstream geometric vision tasks (binocular depth estimation, relative pose estimation, image flow estimation, etc.), the task-specific machine learning model is applied to a pair of input images. The images of an image pair may depict the same visual content (the same scene) and may be taken from different viewpoints or in different conditions (e.g., different focal lengths, different depths of field, different lighting conditions, different seasons).

[0145] The task-specific encoder generates a respective image representation for each input image. In one embodiment, the one or more image representations are input to a task-specific decoder to generate a decoded image representation. The one or more image representations or the decoded image representations may be input to a task-specific output layer (e.g., a 2D convolutional layer or a multi-layer perceptron) of the task-specific machine learning model to generate task-specific output data (e.g., a depth map of the scene, a relative rotation matrix, and a relative deformation vector) according to a geometric downstream vision task. The one or more annotated images are annotated with ground truth data corresponding to the geometric downstream vision task. The value of the task-specific loss function may be determined by the training module 50 based on a metric that quantifies the difference between the generated output data and the ground truth data. The task-specific machine learning model is then tuned by the training module 50, for example, by adjusting a set of task-specific encoder parameters (and optionally a set of task-specific decoder parameters) of the task-specific machine learning model based on or to minimize the value of the task-specific loss function.

[0146] The next iterative stage may begin with or include applying the tuned task-specific machine learning model to one or more annotated images to generate new task-specific output data, determining new values ​​of the task-specific loss function, and further tuning the learnable parameters of the task-specific encoder (and, optionally, the task-specific decoder). This iterative process stops after a predefined number of iterations (e.g., epochs) or when a minimum of the task-specific loss function is reached (when the minimization of the loss function has converged).

[0147] Once the task-specific machine learning model has been fine-tuned for the geometric downstream vision task, the model may be applied to perform the geometric downstream vision task on a set of images. The fine-tuned model is applied to one or more unannotated images in the same manner as in the iterative phase of the fine-tuning phase (e.g., fine-tuning phase 150). However, the determination of the value of the loss function and the iterative adjustment of the task-specific machine learning model are not performed in the application phase.

[0148] As an example, a task-specific machine learning model may be fine-tuned for relative pose estimation. In such an example, the fine-tuned task-specific machine learning model is applied to a new pair of images that are not part of the pre-trained or fine-tuned dataset. The new pair of images depict two views of the same scene (the same scene depicted from different perspectives). By applying the fine-tuned model to the new pair of images, the relative rotation and relative translation between the views of the images in the new pair of images are determined by the task-specific machine learning model as predicted data.

[0149] As an example, the geometric downstream vision task for which the task-specific machine learning model is fine-tuned may be depth estimation. Depth estimation may be performed in a monocular manner (using a single image) and a binocular manner (using two images). In the monocular case, the fine-tuned task-specific machine learning model is applied to a new image (not part of the pre-trained or fine-tuned dataset) to extract a depth map for the new image as predicted data. In the binocular case, the fine-tuned task-specific machine learning model is applied to a pair of new images depicting the same scene from different viewpoints to extract a depth map for the depicted scene as output predicted data.

[0150] As another example, a task-specific machine learning model may be fine-tuned for the estimation of optical flow. For this geometric downstream vision task, the fine-tuned task-specific machine learning model is applied to a new image pair (not part of the pre-trained or fine-tuned dataset) including a new first image and a new second image. The new first image and the new second image depict the same scene under different conditions or from different viewpoints. When applied to the new image pair, the fine-tuned task-specific machine learning model identifies a number of pixel pairs as prediction data. Each pixel pair includes one pixel of the new first image and one corresponding pixel of the new second image, where the pixel pairs correspond to the same visual feature of the depicted scene. The movement of the identified visual feature of the depicted scene between the pixel pairs and the new first image and the new second image is determined by the task-specific machine learning model.

[0151] The pre-learning 510 of FIG. 5 will be described in more detail below with reference to FIGS.

[0152] 6 is a flow chart illustrating a computer-implemented method 600 for pre-training a pre-text machine learning model (e.g., pre-text machine learning model 130) for a pre-text task. Method 600 provides an example implementation of unsupervised pre-training 510 of FIG. 5. The pre-text machine learning model includes an encoder (e.g., encoder 230, 330, or 430) and a decoder (e.g., decoder 270, 370, or 470). The encoder has a set of encoder parameters, and the decoder has a set of decoder parameters.

[0153] Step 610 begins with obtaining an unannotated image pair including a first image (e.g., image 120, 310, or 410) and a second image (e.g., image 122, 320, or 420). The first and second images depict the same scene taken under different conditions (e.g., different focal lengths, different lighting conditions, different seasons) or from different viewpoints.

[0154] At stage 620, an encoder of the pre-text machine learning model encodes a first image into a first representation (e.g., image representation 240, 340, or 440). Additionally, a second image is encoded by the encoder into a second representation (e.g., image representation (250, 350, or 450)).

[0155] At stage 630, the first representation is transformed into a transformed representation (e.g., transformed representation 264, 364, or 468). The transformation may be performed in representation space block 260, cross-view completion block 360, or cross-view alignment block 460. At stage 640, the transformed representation is decoded into a reconstructed image (e.g., reconstructed image 140, 380, or 480) by a decoder of the pre-text machine learning model. The reconstructed image is based on the first and second representations in that either the transformation of the first representation depends on the second representation or the decoder takes both the transformed and second representations as inputs to decode the first representation into a reconstructed image conditioned on the second representation.

[0156] At stage 650, the encoder and decoder are tuned (i.e., updated or modified) by the training module 50 by adjusting each set of encoder and decoder parameters based on or to minimize the loss function (to minimize the value of the loss function (e.g., pre-text loss 142)). By adjusting the encoder and decoder parameters, the pre-text machine learning model is adjusted to better perform the pre-text task after tuning. The pre-text machine learning model may include additional learnable parameters, for example as part of a representation space block, such as the representation space block 260. Such selective additional learnable parameters are also adjusted based on minimizing the loss function. To minimize the loss function, the pre-text loss (value of the loss function) may be determined based on the deviation of the reconstructed image from the image to be reconstructed (e.g., one of the images of the input image pair). Additionally or alternatively, the loss may be determined based on the deviation of the transformed representation from one of the image representations generated by applying the encoder to the image of the input image pair. Adjustment of the learnable parameters (including the encoder and decoder parameters) of the pre-text machine learning model is performed by backpropagation (e.g., backpropagation 144). After adjusting the pre-text machine learning model, the process may be stopped or repeated. If the process is repeated, a new iteration is started at stage 610 with the adjusted pre-text machine learning model including the adjusted encoder and decoder. The process may be stopped after a predetermined number of iterations (e.g., epochs) or when a minimum of the pre-text loss function is reached (e.g., when the value of the pre-text loss function tends to or converges to a minimum).

[0157] 7 is a flow chart illustrating an example of a computer-implemented method 700 for performing cross-view completion pre-training of a pre-text machine learning model (e.g., pre-text machine learning model 130) for a cross-view completion pre-text task. Such method 700 is an example of the pre-training stage 510 of FIG. 5 and is an example of the method 600 for the specific pre-text task of cross-view completion. The pre-text machine learning model includes an encoder (e.g., encoder 330) and a decoder (e.g., decoder 370). The encoder has a set of encoder parameters, and the decoder has a set of decoder parameters.

[0158] Method 700 begins at step 710 with obtaining an image pair including a first image (e.g., image 310) and a second image (e.g., image 320). The first and second images depict the same scene (same visual content) and are taken in different conditions or different viewpoints. Thus, the two images of the image pair represent the same visual content but from different viewpoints, different lighting, different depth of field, different focal lengths, or other differences. In particular, the pixel content of the two images of the image pair differs from one another but depict the same scene. Each image of the image pair provides a distinct view of the depicted scene that is different from the views provided by the other images of the pair.

[0159] At step 720, the first image is divided into a first set of disjoint patches (e.g., image patch 312) and the second image is divided into a second set of disjoint patches (e.g., image patch 322). At step 730, a number of patches in the first set of patches are masked. As an example, a predetermined percentage (e.g., 75-95%) of the patches in the first set of patches are randomly selected to be masked.

[0160] After splitting the first image, at stage 740, the encoder encodes the first image into a first representation (e.g., image representation 340) by encoding the remaining unmasked patches of the first set of patches into corresponding unmasked patch representations (e.g., patch representation 342) to generate a first set of patch representations. Encoding the first image into the first representation includes encoding only the unmasked patches into the first set of patch representations such that the first set of patch representations does not include patch representations for masked patches (e.g., masked patch 314).

[0161] At step 750, the encoder encodes the second image into a second representation (e.g., image representation 350) by encoding each patch in the second set of patches into a corresponding respective patch representation (e.g., patch representation 352) to generate a second set of patch representations. Steps 740 and 750 are an example implementation for the CroCo pre-text task shown in step 620 of FIG. 6.

[0162] At step 760, the first representation of the first image is transformed into a transformed representation (e.g., transformed representation 364). As an example, the first representation is transformed into the transformed representation by patching, for each masked patch of the first image, a first patch representation set including only patch representations for unmasked patches of the first image with a respective learned representation (e.g., learned representation 362). The resulting transformed representation may include each of the patch representations of the first patch representation set (e.g., patch representation 342) and include multiple learned patch representations corresponding to the masked patches (e.g., learned patch representation 362).

[0163] Following transformation of the first representation, the method proceeds to decoding, by a decoder, the transformed representation into a reconstructed image (e.g., reconstructed image 380) (step 770). The transformed representation is decoded by generating, for each masked patch of the first set of patches, a predicted reconstruction of the respective masked patch based on the first and second patch representation sets. In one example, generating the predicted reconstruction of the masked patches of the first set of patches includes decoding, by the decoder, the learned representation of the masked patch into the predicted reconstruction of the masked patch. To decode the learned representation of the masked patch, the decoder receives the first and second patch representation sets as input data and decodes the learned representation of the masked patch based on the input data. In one embodiment, the transformed representation includes patch representations of the first patch representation set, and the transformed representation is decoded into a reconstructed image conditioned on the second representation.

[0164] In stage 780, the encoder and decoder are tuned (e.g., updated or modified) by adjusting a set of corresponding encoder and decoder parameters for minimization of a loss function (e.g., minimization of the value of the loss function (e.g., pre-text loss 142)). In one example, the loss function may be based on a metric that quantifies the difference between each masked patch of the first image and the respective predicted reconstruction. In one example, the pre-text machine learning model may include a set of learnable parameters of a learned representation corresponding to the masked patch of the first image. In this example, the additional learnable parameters and the corresponding learned representation are also tuned to minimize the loss function. By tuning the encoder and decoder parameters, and optionally the parameters of the learned representation, the pre-text machine learning model is tuned such that it performs the cross-view completion task better after tuning. The tuning of the learnable parameters (including the encoder and decoder parameters) of the pre-text machine learning model is performed by backpropagation (e.g., backpropagation 144). After tuning the pre-text machine learning model, the process may be stopped or may be repeated. If the process is to be repeated, a new iteration is started with the tuned pre-text machine learning model, including the tuned encoder and decoder (and optionally the tuned learnable representation), by 710. The method may be stopped after a predetermined number of iterations are completed or when a minimum of the pre-text loss function is reached (e.g., when the value of the pre-text loss function converges toward or to a minimum).

[0165] 8 is a flow chart illustrating an example of a computer-implemented method 800 of pre-training 510 of FIG. 5, i.e., a computer-implemented method for performing cross-view alignment pre-training of a pre-text machine learning model (e.g., pre-text machine learning model 130) for a cross-view alignment pre-text task. Method 800 is an example of method 600 for the specific pre-text task of cross-view alignment. The pre-text machine learning model includes an encoder (e.g., encoder 430) and a decoder (e.g., decoder 470). The encoder has a set of encoder parameters, and the decoder has a set of decoder parameters.

[0166] The method begins by obtaining an image pair including a first image (e.g., source image 410) and a second image (e.g., target image 420), where the first and second images depict the same scene and are taken in different conditions or from different viewpoints. Thus, the two images of the image pair represent the same visual content but from different viewpoints or lighting, different depth of field, different focal lengths, or other visual differences relative to the same scene. In particular, the pixel content of the two images of the image pair differs from each other but contains the same scene. Each image of the image pair provides a distinct view of the depicted scene that differs from the views provided by the other images of the pair.

[0167] At stage 820, an encoder of the pre-text machine learning model encodes the first image into a first representation of the first image (e.g., source representation 440) and encodes the second image into a second representation of the second image (e.g., target representation 450). As an example, the first representation may be a first set of n vectors (e.g., an aligned set):

number

number

[0168] At step 830, the first representation is transformed into a transformed representation (e.g., transformed source representation 468) by applying a transformation (e.g., transformation 466) to the first representation. The transformation is determined based on the first and second representations such that the transformed representation approximates (e.g., aligns) with the second representation. The transformation depends on a set of transformation parameters (e.g., parameters (Ω) 464) determined by a parameter module (e.g., parameter module 462) based on the first and second representations. In one embodiment, each vector in the first and second vector sets is decomposed into a D-dimensional equilateral part and a (KD)-dimensional invariant part, where D is a natural number between 1 and K. In this example, applying the transformation includes decomposing each vector and applying a (D×D)-dimensional transformation matrix (Ω) to the equilateral part of each vector in the first vector set. Applying the transformation to any vector in the first and second vector sets does not change the invariant part of the respective vector. Applying the transformation matrix (Ω) to the equilateral portions of the vectors may be performed by multiplying the transformation matrix (Ω) with the equilateral portions of the vectors. The transformation matrix (Ω) may be determined to align the equilateral portions of the vectors of the first vector set with the equilateral portions of each vector of the first vector set. In one embodiment, the transformation may be a D-dimensional rotation and the transformation matrix (Ω) may be a D-dimensional rotation matrix. For such an embodiment, the transformation matrix (Ω) may be determined by the following equation:

number

[0169] where x i,i equiv is a vector (x i,j ) and denote the equilateral part of x 2,i equiv is a vector (x 2,i) where SO(D) denotes the equilateral part of the D-dimensional rotation group (i.e., the special orthogonal group of dimension D). In the above formula, the function "arg" indicates that the transformation value is not the minimum of the convolution, but the matrix (^Ω) for which the minimum of the convolution is achieved.

[0170] At step 840, the decoder decodes the transformed representation into a reconstructed image (eg, reconstructed image 480).

[0171] At stage 850, the encoder and decoder are tuned (i.e., updated or modified) by adjusting a set of corresponding encoder and decoder parameters for minimization of a loss function (minimization of the value of the loss function (pre-text loss 142)). The loss function is based on a metric that quantifies the difference between the reconstructed image and the second image. Additionally or alternatively, the loss function may be based on a metric that quantifies the difference between the transformed representation and the second representation. By adjusting the encoder and decoder parameters, the pre-text machine learning model is adjusted so that after the adjustment, it performs the cross-view alignment task better. The adjustment of the learnable parameters (including the encoder and decoder parameters) of the pre-text machine learning model may be performed by backpropagation (e.g., backpropagation 144). After tuning the pre-text machine learning model, the process may be stopped or may be repeated. If the process is repeated, a new iteration starts at stage 810 with the adjusted pre-text machine learning model including the adjusted encoder and decoder. The process may be stopped after a predetermined number of iterations are completed or when a minimum of the pre-text loss function is reached (e.g., when the value of the pre-text loss function converges toward or to a minimum).

[0172] The two pre-text tasks, cross-view completion and cross-view alignment, may be combined to train a task-specific machine learning model for the geometric downstream vision task. To this end, various techniques of combining methods may be implemented. As an example (called the "early fusion approach"), two separate pre-text machine learning models are pre-trained by the training module 50. The first pre-text machine learning model is pre-trained by the cross-view completion pre-training method 700. The second pre-text machine learning model is pre-trained by the cross-view alignment pre-training method 800. This results in two pre-trained encoders, one pre-trained for the CroCo pre-text task and the other pre-trained for the Caiman pre-text task. A task-specific machine learning model (e.g., model 170) for the geometric downstream vision task is built by the training module 50 based on or using both such pre-trained encoders. As an example, guidance fine-tuning (e.g., supervised fine-tuning 150) is performed by the training module 50 by feeding one or more input images to a first pre-trained encoder to yield one or more first representations for the one or more images. Similarly, the one or more input images are fed by the training module 50 to a second pre-trained encoder, which yields one or more representations for the one or more input images. A combined representation is then generated by the training module 50 for each of the one or more input images by concatenating the first and second representations for each image. Finally, the combined representation is input to a final task-specific layer (e.g., a multi-layer perceptron) of the task-specific machine learning model. The task-specific layer generates task-specific output data according to a geometric downstream vision task, and a task-specific loss (e.g., loss 182) is determined by the training module 50 based on the output data and the annotated ground truth data for the one or more input images.Finally, the learnable parameters of the task-specific machine learning model (including the parameters of the encoder and the parameters of the task-specific layers) are adjusted by training module 50, for example, by backpropagation (eg, backpropagation 184).

[0173] Another example for combining cross-view completion pre-training and cross-view alignment pre-training (referred to as the "late fusion approach") is shown in Figure 9, which is a flowchart illustrating a computer-implemented method 900 for generating prediction data for a geometric downstream vision task. The method 900 begins at step 910 with training a first task-specific machine learning model (e.g., model 170) for a geometric downstream vision task using cross-view alignment pre-training of a first pre-text machine learning model (e.g., model 130). The training of the first task-specific machine learning model is performed according to method 500, and the pre-training step 510 is performed according to cross-view alignment pre-training method 800.

[0174] In step 920, a second task-specific machine learning model is trained for the geometric downstream vision task using the cross-view completion pre-training of the second pre-text machine learning model. The training of the second task-specific machine learning model is performed by method 500, and the pre-training step 510 is performed by cross-view completion pre-training method 700. Steps 910 and 920 may be performed in sequence according to any order, or in parallel.

[0175] In steps 930 and 940, two independently trained task-specific machine learning models are applied to at least one image to extract respective predicted data from the at least one image. Steps 930 and 940 may be performed in sequence according to any order or in parallel. Specifically, in step 930, first predicted data according to a geometric downstream vision task is generated by applying the trained first task-specific machine learning model to the at least one image. In step 940, second predicted data according to a geometric vision task is generated by applying the trained second task-specific machine learning model to the at least one image. In step 950, a first confidence value is determined for the first predicted data, and a second confidence value is determined for the second predicted data. The first confidence value may indicate the accuracy of the prediction (e.g., estimation) of the first predicted data for the geometric downstream vision task, and the second confidence value may indicate the accuracy of the prediction (e.g., estimation) of the second predicted data for the geometric downstream vision task. The first and second confidence values ​​may be determined by applying the respective trained task-specific machine learning models to a test set of annotated images and comparing the resulting extracted predicted data of the annotated images of the test set with the annotated ground truth data of the annotated images of the test set. As shown in FIG. 9, stage 950 may be performed after stages 930 and 940 and before stage 960. However, stage 950 may be performed at any time after stages 910 and 920 and before 960. In particular, stage 950 may be performed before stages 930 and 940 or between stages 930 and 940.

[0176] The method concludes with generating resulting predicted data for the geometric vision task by fusing the first predicted data and the second predicted data based on the first confidence value and the second confidence value at step 960. Fusing the first predicted data and the second predicted data includes determining a weighted sum of the predicted data, where the weights are based on the respective confidence values.

[0177] Although the above examples have been described in the context of a method, this is also a description of corresponding components, modules or features of a corresponding apparatus or system. Some or all of the functionality of the methods may be computer-implemented in that they are performed by (or using) one or more processors, microprocessors, electronic circuits, and / or processing circuits.

[0178] The methods and features described above may be implemented in an architecture such as that shown in the functional block diagram of Figure 10, which includes a server 1000 and one or more computing devices (collectively 1002) that communicate over a network 1004 (wireless and / or wired), such as the Internet, for data exchange. The server 1000 and the computing devices 1002 each include one or more processors 1012 and a memory 1013, such as a hard disk. The computing devices 1002 may include any computing device that communicates with the server 1000, such as an autonomous vehicle 1002b, a robot 1002c, a computer 1002d or a mobile phone 1002e, or any other suitable type of computing device.

[0179] As an example, the method 500 for training a task-specific machine learning model for a geometric downstream vision task is performed in combination with the pre-learning method 600, 700, or 800. The server may provide the trained task-specific machine learning model to any server device, which applies the trained machine learning model to one or more images inputted to extract predictive data according to the geometric downstream vision task from one or more images. As an example, one of the devices performs the method 500 for training a task-specific machine learning model for a geometric downstream vision task in combination with the pre-learning method 600, 700, or 800, and then applies the trained task-specific machine learning model to one or more images to extract predictive data according to the geometric downstream vision task from one or more images. In one embodiment, one of the devices performs the method 900 for generating predictive data according to a geometric downstream vision task.

[0180] In additional embodiments, an autonomous device (e.g., vehicle 1002b or robot 1002c) may include an optical sensor (e.g., a camera) that generates a first image of the device's surroundings. The autonomous device may further include a second optical sensor (e.g., a second camera) that generates a second image of the device's surroundings. The first image and the second image depict the device's surroundings from different perspectives. Alternatively, the autonomous device does not include a second optical sensor and records a second image of the device's surroundings in a memory device (e.g., memory device 1013b / c). The recorded second image may be previously generated by the optical sensor. The autonomous device may train a task-trained machine learning model for a geometric downstream vision task by method 500 in combination with pre-learning method 600, 700, or 800. Alternatively, the autonomous device may receive the trained task-specific machine learning model from a server, which trains the task-specific machine learning model for a geometric downstream vision task by method 500 in combination with pre-learning method 600, 700, or 800. The autonomous device may apply the trained task-specific machine learning model to one or more images to extract predictive data from one or more images by a geometric vision task. The autonomous device may further adjust its moving state (e.g., speed or moving direction) or its operation based on the extracted predictive data. That is, the autonomous device may adjust speed and / or steering based on the extracted predictive data.

[0181] In one embodiment, the geometric downstream vision task for which the task-specific machine learning model was trained may be monocular depth estimation. The autonomous device applies the trained task-specific machine learning model to a first image, thereby extracting a depth map from the first image, where the depth map corresponds to the autonomous device's surroundings. The depth map may include an indication for each pixel or patch of a respective relative distance from the pixel or patch to the nearest object with respect to the camera that captured the image. Based on the extracted depth map, the autonomous device determines a distance to an object present in the autonomous device's surroundings. The autonomous device adjusts its speed and / or direction of movement based on the determined distance to the object.

[0182] In another embodiment, the geometric downstream vision task for which the task-specific machine learning model was trained may be binocular depth estimation. The autonomous device applies the trained task-specific machine learning model to the image pair, thereby extracting a depth map from the image pair, where the depth map corresponds to the autonomous device's surroundings. The depth map includes an indication for each pixel or patch of a respective relative distance from the pixel or patch to a nearest object with respect to a camera that captured the image. The image pair includes a first image generated by an optical sensor and a second image generated by a second optical sensor, where the images of the image pair are generated by the optical sensors substantially simultaneously. Based on the extracted depth map, the autonomous device determines a distance to an object present in the autonomous device's surroundings. Based on the determined distance to the object, the autonomous device adjusts its speed and / or its direction of movement. That is, the autonomous device adjusts its speed and / or steering based on the extracted distance data.

[0183] In yet another embodiment, the geometric downstream vision task for which the task-specific machine learning model was trained may be relative pose estimation. The autonomous device applies the trained task-specific machine learning model to a pair of images, thereby extracting a relative rotation matrix and a relative deformation vector from the pair of images. The pair of images includes a first image generated by an optical sensor and a second image recorded in the memory of the device, where the second image was generated by the optical sensor or another optical sensor prior to the first image. The memory of the device may further record position information of the second image. The position information may indicate a previous position of the device relative to the surroundings of the device. Based on the extracted relative rotation matrix, relative deformation vector, and position information, the autonomous device determines a new position of the device relative to the surroundings of the autonomous device. The autonomous device adjusts its speed and / or its direction of movement based on the determined new position. That is, the autonomous device adjusts its speed and / or steering based on the determined position.

[0184] In yet another embodiment, the geometric downstream vision on which the task-specific machine learning model is trained may be an estimation of an optical flow quantity. The autonomous device applies the trained task-specific machine learning model to a pair of images, thereby extracting corresponding pixel pairs for the pair of images. The pair of images includes a first image generated by an optical sensor and a second image stored in the memory of the device, where the second image was generated by the optical sensor prior to the first image. The extracted corresponding pixel pair includes one pixel from the first image and one corresponding pixel from the second image. The autonomous device determines a speed and / or a moving direction of an object present in the autonomous vehicle's surroundings based on the extracted corresponding pixel pair. The autonomous vehicle adjusts its speed and / or its moving direction based on the determined speed and / or moving direction of the object. That is, the autonomous device adjusts its speed and / or steering based on the determined speed and / or moving direction of the object, for example, to avoid a collision with the object.

[0185] The above description is merely exemplary in nature and is not intended to limit the disclosure, its application, or uses. The broad teachings of the present disclosure can be realized in various forms. Thus, while the present disclosure includes certain embodiments, other modifications will become apparent from a study of the drawings, the specification, and the appended claims, and the actual scope of the present disclosure should not be limited thereto. It should be understood that one or more steps included in the above-described method can be performed in different orders (or simultaneously) without changing the principles of the present disclosure. Also, although each embodiment has been described as having specific features, one or more features described in connection with one embodiment of the present disclosure may be implemented as features of any of the other embodiments, or may be combined with features of any of the other embodiments, even if the combination is not expressly described. That is, the above-described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other are maintained within the scope of the present disclosure.

[0186] Spatial and functional relationships between elements (e.g., modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "interdigitated," "coupled," "adjacent," "next to," "above," "above," and "disposed." Unless expressly stated as "direct," when describing a relationship between a first element and a second element in this disclosure, the relationship may be a direct relationship where there are no other intervening elements between the first element and the second element, but may also be an indirect relationship where there are one or more intervening elements (spatial or functional) between the first element and the second element. As used herein, the terms at least one of A, B, and C should be interpreted as meaning a logic (A OR B OR C) using a non-exclusive logical OR, and not as "at least one of A, at least one of B, and at least one of C."

[0187] In a drawing, the direction of an arrow generally indicates the flow of information (such as data or commands) of interest within the drawing. For example, if element A and element B are in a relationship in which they exchange various information, and the information sent from element A to element B is relevant to the drawing in question, the arrow is shown pointing from element A to element B. This unidirectional arrow does not mean that no information is sent from element B to element A. Also, in the case of information sent from element A to element B, element B may send element A a request for the information or an acknowledgment of receipt of the information.

[0188] In this application, including the definitions below, the terms "module" or "controller" may be interchangeable with the term "circuitry." The term "module" may mean, be a part of, or include, any or all combinations thereof, such as an Application Specific Integrated Circuit (ASIC), digital, analog, or mixed analog / digital discrete circuitry, digital, analog, or mixed analog / digital integrated circuit, combinatorial logic circuitry, Field Programmable Gate Array (FPGA), processor circuitry (shared, dedicated, or group) that executes code, memory circuitry (shared, dedicated, or group) that stores code executed by the processor circuitry, other suitable hardware components that provide the functionality described above, or a system on a chip, etc.

[0189] A module may include one or more interface circuits. As an example, the interface circuit may include a wired or wireless interface that connects to a LAN, the Internet, a WAN, or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing. As another example, a server (also called a remote or cloud) module may perform some functions on behalf of the module.

[0190] As discussed above, code may include software, firmware, and / or microcode and may refer to programs, routines, functions, classes, data structures, and / or objects. A shared processor circuit includes a single processor circuit that executes some or all of the code from multiple modules. The term group processor circuit includes a processor circuit that executes some or all of the code from one or more modules in combination with additional processor circuits. References to multiprocessor circuits include multiple processor circuits on separate dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations thereof. A shared memory circuit includes a single memory circuit that stores some or all of the code from multiple modules. A group memory circuit includes a memory circuit that stores some or all of the code from one or more modules in combination with additional memory.

[0191] A memory circuit is a subset of computer-readable media. As used herein, the term computer-readable media does not include a transitory electric or electrical signal propagated by a medium (e.g., a carrier wave). Thus, the term computer-readable media is considered to be tangible and non-transitory. Non-limiting examples of tangible and non-transitory computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable dedicated memory circuits, or masked read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic recording media (e.g., analog or digital magnetic tape, or hard disk drive), and optical recording media (e.g., CD, DVD, or Blu-ray disk).

[0192] The apparatus and method described in this application may be partially or completely implemented by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions implemented by a computer program. The above-described functional blocks, flow chart components, and other elements function as software that can be converted into a computer program by the routine work of a skilled technician or programmer.

[0193] A computer program includes processor-executable instructions recorded on at least one tangible, non-transitory computer-readable medium. A computer program includes or relies on recorded data. A computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0194] A computer program may include (i) parsed explanatory text, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a compiler in an opportunistic manner; and the like. For example, source code may be created using syntax from languages ​​including C, C++, C#, ObjectiveC, Swift, Haskell, Go, SQL, R, Lisp, Java, Fortran, Perl, Pascal, Curl, OCaml, Javascript, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash, Visual Basic, Lua, MATLAB, SIMULINK, and Python.

Claims

1. 1. A computer-implemented machine learning method for training a task-specific machine learning model for a geometric downstream vision task, comprising: The method comprises: performing unsupervised pre-training of a machine learning model including an encoder having a set of encoder parameters and a decoder having a set of decoder parameters; constructing a task-specific machine learning model for the geometric downstream vision task based on the pre-trained machine learning model, the task-specific machine learning model including a task-specific encoder having a set of task-specific encoder parameters; initializing the set of task-specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model; fine-tuning the task-specific machine learning model initialized with the set of encoder parameters for the geometric downstream vision task. Including, The step of performing the unsupervised pre-training of the machine learning model includes: acquiring a pair of unannotated images, the first and second images depicting the same scene and taken under different conditions or from different viewpoints; encoding, by the encoder, the first image into a first image representation and the second image into a second image representation; converting the first image representation into a transformed representation; decoding, by said decoder, said transformed representation into a reconstructed image; and adjusting at least one parameter of at least one of the encoder and the decoder based on minimizing loss. Including, the steps of transforming the first image representation and decoding the transformed representation are based on the first image representation and the second image representation. method.

2. The unsupervised pre-training of the machine learning model is cross-view alignment pre-training; Transforming the first image representation comprises: applying a transformation to the first image representation to generate the transformed representation. Including, the transformation is determined based on the first image representation and the second image representation such that the transformed representation approximates the second image representation. The method of claim 1.

3. The unsupervised pre-training of the machine learning model is cross-view alignment pre-training, the loss is based on a metric that quantifies a difference between the reconstructed image and the second image. The method of claim 1.

4. The unsupervised pre-training of the machine learning model is cross-view alignment pre-training, the loss is based on a metric that quantifies a difference between the transformed representation and the second image representation. The method of claim 1.

5. The first image representation is a first set of n vectors: [0010] and The second image representation is a second set of n vectors: [0025] and The step of applying the transformation comprises: decomposing each vector of the first set of n vectors and the second set of n vectors in a D-dimensional equilateral part and a (K-D)-dimensional invariant part; and applying a (D×D) dimensional transformation matrix (Ω) to the equilateral portion of each vector of the first set of n vectors. Including, 0<D≦K; The method of claim 2.

6. the transformation is a D-dimensional rotation, and the transformation matrix (Ω) is a D-dimensional rotation matrix; the transformation matrix (Ω) is established based on aligning equilateral portions of vectors of the first set of n vectors with equilateral portions of each vector of the second set of n vectors. The method according to claim 5.

7. determining the transformation matrix (Ω) based on the following equation: [0030] Further comprising: Here, x ｉ，ｉ ｅｑｕｉｖ is a vector x ｉ，ｊ Denote the equilateral part of x ２，ｉ ｅｑｕｉｖ is a vector x ２，ｉ and SO(D) denotes the D-dimensional rotation group. The method according to claim 5.

8. The unsupervised pre-training is cross-view completion pre-training, performing the cross-view complete pre-training of the machine learning model; partitioning the first image into a first set of disjoint patches and partitioning the second image into a second set of disjoint patches; masking at least one of the patches of said first set of disjoint patches. Further comprising: The step of encoding the first image into the first image representation comprises: encoding, by the encoder, each unmasked patch of the first set of disjoint patches into a corresponding unmasked patch representation to generate a first set of patch representations; Including, Encoding the second image into the second image representation comprises: encoding, by the encoder, each patch in the second set of disjoint patches into a corresponding respective patch representation to generate a second set of patch representations; Including, The step of decoding the transformed representation comprises: generating, by the decoder, for each masked patch of the first set of disjoint patches, a predicted reconstruction for the respective masked patch based on the transformed representation and the second set of patch representations; Including, The loss function is based on a metric that quantifies the difference between each masked patch and its respective predicted reconstruction. The method of claim 1.

9. Transforming the first image representation into the transformed representation comprises: for each masked patch of the first set of disjoint patches, patching the first set of patch representations with the respective learned representation of the masked patch. Further comprising: The method according to claim 8.

10. Each learned representation includes a set of representation parameters, 10. The method of claim 9.

11. The step of generating a predicted reconstruction of a masked patch of the first set of disjoint patches comprises: decoding, by the decoder, the learned representation of the masked patch into the predicted reconstruction of the masked patch; Including, wherein the decoder receives the first and second patch representation sets as input data, and decodes the learned representations of the masked patches based on the input data; The method comprises: adjusting the learned representations of the masked patches by adjusting a respective set of representation parameters. Further comprising:

10. The method of claim 9.

12. adjusting the respective sets of representation parameters includes adjusting the sets of representation parameters based on minimizing the loss. The method of claim 11.

13. The step of fine-tuning the task-specific machine learning model includes: performing supervised fine-tuning of the task-specific machine learning model. Including, The method of claim 1.

14. The supervised fine-tuning step includes: i) applying the task-specific machine learning model to one or more annotated images that are annotated as ground truth data for the geometric downstream vision task to generate output data according to the geometric downstream vision task; and ii) tuning the task-specific machine learning model by adjusting a set of task-specific encoder parameters of the task-specific machine learning model based on minimizing a task-specific loss based on a metric that quantifies a difference between the generated output data and ground truth data. The method of claim 13, comprising:

15. repeating i) and ii) a predetermined number of times or until a minimum value of said task specific loss is reached. The method of claim 14 further comprising:

16. applying the fine-tuned task-specific machine learning model to one or more new images to extract predictive data from the one or more new images for a geometric downstream vision task. The method of claim 14 further comprising:

17. The geometric downstream vision task is relative pose estimation.

17. The method of claim 16.

18. 2. The step of applying the fine-tuned task-specific machine learning model, applying the fine-tuned task-specific machine learning model to new image pairs to extract relative rotations and relative transformations between images of the new image pairs as predicted data. Including, the images of the new image pair depict two views of the same scene; 20. The method of claim 17.

19. The geometric downstream vision task is depth estimation.

17. The method of claim 16.

20. 2. The step of applying the fine-tuned task-specific machine learning model, applying the fine-tuned task-specific machine learning model to one or more new images to extract one or more depth maps from the one or more new images as predicted data.

20. The method of claim 19, comprising:

21. The geometric downstream vision task is the estimation of optical flow.

17. The method of claim 16.

22. 2. The step of applying the fine-tuned task-specific machine learning model, applying the fine-tuned task-specific machine learning model to a new image pair including a new first image and a new second image to generate a plurality of pixel pairs as the prediction data. Including, each pixel pair includes a pixel of the new first image and a corresponding pixel of the new second image; the new first image and the new second image depict the same scene under different conditions or from different viewpoints; 22. The method of claim 21.

23. 1. A computer-implemented machine learning method for generating predictive data via a geometric downstream vision task, comprising: The method comprises: training a first task-specific machine learning model for the geometric downstream vision task using cross-view alignment pre-training of the first machine learning model; training a second task-specific machine learning model for the geometric downstream vision task using the cross-view completion pre-training of a second pre-text machine learning model; generating first prediction data according to the geometric downstream vision task by applying the trained first task-specific machine learning model to at least one image; generating second predicted data according to the geometric downstream vision task by applying the trained second task-specific machine learning model to the at least one image; determining a first confidence value for the first predicted data and a second confidence value for the second predicted data; and generating a resultant predicted data according to the geometric downstream vision task by fusing the first predicted data and the second predicted data together based on the first confidence value and the second confidence value. Includes, a method.

24. 1. A computing system comprising: one or more processors; and a memory comprising code that, when executed by the one or more processors, performs the following: Including, The code is performing unsupervised pre-training of a machine learning model including an encoder having a set of encoder parameters and a decoder having a set of decoder parameters; constructing a task-specific machine learning model for a geometric downstream vision task based on the pre-trained machine learning model, the task-specific encoder having task-specific encoder parameters; initializing the set of task-specific encoder parameters with the set of encoder parameters of the pre-trained machine learning model; fine-tuning the task-specific machine learning model initialized with the set of encoder parameters for the geometric downstream vision task; The unsupervised pre-training of the machine learning model is performed by: obtaining a pair of unannotated images including a first image and a second image depicting the same scene and taken under different conditions or from different viewpoints; encoding, by the encoder, the first image into a first image representation and the second image into a second image representation; converting the first image representation into a transformed representation; decoding, by said decoder, said transformed representation into a reconstructed image; adjusting at least one parameter of at least one of the encoder and the decoder based on minimizing loss; Including, transforming the first image representation and decoding the transformed representation are based on the first image representation and the second image representation. Computing system.

Images