Computer vision training system and method for training a computer vision system

Abstract

A computer vision training system and a method for training a computer vision system are disclosed. The computer vision (CV) training system includes: a supervised learning system that estimates a supervised output from one or more input images according to a target CV application and determines a supervised loss based on the supervised output and a ground truth of the supervised output; an unsupervised learning system that determines an unsupervised loss based on the supervised output and the one or more input images; a weakly supervised learning system that determines a weakly supervised loss based on the supervised output and weak labels corresponding to the one or more input images; and a joint optimizer that concurrently optimizes the supervised loss, the unsupervised loss, and the weakly supervised loss.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 927,625, filed on October 29, 2019, entitled “SYSTEM AND METHOD FOR PROVIDING A DEEPLEARNING FRAMEWORK BASED ON MULTIPLE LEARNING TECHNIQUES FOR COMPUTER VISION APPLICATION,” and also claims priority to and the benefit of U.S. Provisional Application No. 62 / 966,975, filed on January 28, 2020, entitled “DEEP LEARNING FRAMEWORK BASED ON MULTIPLE LEARNING STRATEGIES FOR CV APPLICATIONS WITH MULTI-FRAME INPUT,” all of which are incorporated herein by reference in their entirety. Technical Field

[0002] One or more example embodiments of the present disclosure relate to a computer vision system and a method of training the computer vision system using machine learning. Background Art

[0003] Computer vision applications, such as depth estimation and image / video enhancement, are widely used in real-world scenarios. For example, depth estimation can be used for foreground / background segmentation of image bokeh, and image / video enhancement has become a standard feature in many image signal processors (ISPs) in most modern cell phone cameras.

[0004] The field of computer vision has recently shifted from statistical methods to deep learning neural network methods. For example, deep learning neural network methods enable a single model to learn meaning from images to perform a specific computer vision task, thus avoiding the need for a pipeline of specialized handcrafted methods. Deep learning neural network methods often utilize a single machine learning technique and corresponding loss function trained for a specific computer vision application, which can limit their accuracy.

[0005] The above information disclosed in this Background section is for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute prior art. Summary of the Invention

[0006] One or more example embodiments of the present disclosure relate to a computer vision system having a multiple machine learning strategy framework for training for various computer vision applications, and a method for training a computer vision system.

[0007] According to one or more example embodiments of the present disclosure, a computer vision (CV) training system includes: a supervised learning system, configured to estimate a supervised output from one or more input images according to a target CV application, and determine a supervised loss based on the supervised output and a ground truth value of the supervised output; an unsupervised learning system, configured to determine an unsupervised loss based on the supervised output and the one or more input images; a weakly supervised learning system, configured to determine a weakly supervised loss based on the supervised output and weak labels corresponding to the one or more input images; and a joint optimizer, configured to optimize the supervised loss, the unsupervised loss, and the weakly supervised loss.

[0008] In one example embodiment, the target CV application may correspond to monocular depth estimation, and the supervised output may correspond to a depth estimated from a target image among the one or more input images.

[0009] In an example embodiment, the unsupervised learning system may be configured to convert 2-dimensional (2D) image coordinates corresponding to the one or more input images into 3-dimensional (3D) coordinates according to the estimated depth, and calculate an unsupervised loss based on at least the 3D coordinates.

[0010] In one example embodiment, the target CV application may correspond to stereo matching, and the supervised output may correspond to an estimated disparity between a left image and a right image among the one or more input images.

[0011] In one example embodiment, the unsupervised learning system may be configured to shift one of the left image and the right image according to the estimated disparity to generate a deformed image, and calculate the unsupervised loss based on at least the deformed image.

[0012] In one example embodiment, the target CV application may correspond to image / video enhancement, and the supervised output may correspond to a high-resolution image estimated from a low-resolution input image among the one or more input images.

[0013] In one example embodiment, the unsupervised learning system may be configured to estimate an optical flow corresponding to the estimated high-resolution image, and calculate the unsupervised loss based on at least the estimated optical flow and the estimated high-resolution image.

[0014] In one example embodiment, the target CV application may correspond to multi-view depth estimation, and the supervision output may correspond to the estimated depth in each of a plurality of input frames among the one or more input images.

[0015] In an example embodiment, the unsupervised learning system may be configured to generate a 3D point cloud for each of the multiple input frames based on the estimated depth in each of the multiple input frames, and calculate an unsupervised loss based at least on a distance between aligned coordinates of the 3D point clouds.

[0016] In an example embodiment, the joint optimizer may be configured to concurrently optimize the supervised loss, the unsupervised loss, and the weakly supervised loss according to a joint loss function, the joint loss function including respective weights of the supervised loss, the unsupervised loss, and the weakly supervised loss.

[0017] In an example embodiment, the system may further include: a weak label generator, the weak label generator being configured to generate weak labels as image block-based weak labels, the image block-based weak labels having image block pairs corresponding to different cropped regions of the one or more input images, the image block-based weak labels defining a relationship corresponding to the supervisory output between the cropped regions of the image block pairs.

[0018] According to one or more example embodiments of the present disclosure, a method for training a computer vision (CV) system includes: estimating a supervised output from one or more input images according to a target CV application; determining a supervised loss based on the supervised output and a ground truth of the supervised output; determining an unsupervised loss based on the supervised output and the one or more input images; determining a weakly supervised loss based on the supervised output and weak labels corresponding to the one or more input images; and optimizing the supervised loss, the unsupervised loss, and the weakly supervised loss.

[0019] In one example embodiment, the target CV application may correspond to monocular depth estimation, and the supervised output may correspond to a depth estimated from a target image among the one or more input images.

[0020] In an example embodiment, the method may further include: converting 2-dimensional (2D) image coordinates corresponding to the one or more input images into 3-dimensional (3D) coordinates according to the estimated depth; and calculating an unsupervised loss based on at least the 3D coordinates.

[0021] In one example embodiment, the target CV application may correspond to stereo matching, and the supervision output may correspond to an estimated disparity between a left image and a right image among the one or more input images.

[0022] In an example embodiment, the method may further include: shifting one of the left image and the right image according to the estimated disparity to generate a deformed image; and calculating the unsupervised loss based on at least the deformed image.

[0023] In one example embodiment, the target CV application may correspond to image / video enhancement, and the supervised output may correspond to a high-resolution image estimated from a low-resolution image among the one or more input images.

[0024] In an example embodiment, the method may further include: estimating an optical flow corresponding to the estimated high-resolution image; and calculating an unsupervised loss based on at least the estimated optical flow and the estimated high-resolution image.

[0025] In one example embodiment, the target CV application may correspond to multi-view depth estimation, and the supervision output may correspond to the estimated depth in each of a plurality of input frames among the one or more input images.

[0026] In an example embodiment, the method may further include: generating a 3D point cloud for each of the multiple input frames based on the estimated depth in each of the multiple input frames; and calculating an unsupervised loss based at least on a distance between aligned coordinates of the 3D point clouds.

[0027] In an example embodiment, the step of optimizing includes concurrently optimizing a supervised loss, an unsupervised loss, and a weakly supervised loss according to a joint loss function, the joint loss function including respective weights of the supervised loss, the unsupervised loss, and the weakly supervised loss.

[0028] In an example embodiment, the method may further include: generating the weak label as an image block-based weak label, the image block-based weak label having an image block pair corresponding to different cropped regions of the one or more input images, and the image block-based weak label definition corresponding to the supervisory output between the cropped regions of the image block pair. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects and features of the present disclosure will become more apparent to those skilled in the art from the following detailed description of example embodiments with reference to the accompanying drawings.

[0030] Figure 1 is a block diagram of a computer vision training system according to one or more example embodiments of the present disclosure.

[0031] Figure 2 is a diagram of a multi-learning strategy framework of a computer vision training system according to one or more example embodiments of the present disclosure.

[0032] Figure 3 is a diagram of a multi-learning strategy framework of a computer vision training system for monocular depth estimation according to one or more example embodiments of the present disclosure.

[0033] Figure 4is a diagram of a multi-learning strategy framework of a computer vision training system for stereo matching according to one or more example embodiments of the present disclosure.

[0034] Figures 5A to 5B is a diagram of a multi-learning strategy framework of a computer vision training system for image / video enhancement according to one or more example embodiments of the present disclosure.

[0035] Figure 6 is a diagram of a multi-learning strategy framework of a computer vision training system for multi-view depth estimation according to one or more example embodiments of the present disclosure.

[0036] 7A to 7B is a table illustrating an overview of a multi-learning strategy framework of a computer vision training system applied to various computer vision applications according to one or more example embodiments of the present disclosure.

[0037] Figure 8 is a flowchart of a method of training a computer vision training system according to one or more example embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings, wherein the same reference numerals represent the same elements throughout. However, the present disclosure can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that the present disclosure will be thorough and complete and will fully convey the aspects and features of the present disclosure to those skilled in the art. Therefore, processes, elements and techniques that are not necessary for those skilled in the art to fully understand the aspects and features of the present disclosure may not be described. Unless otherwise stated, the same reference numerals represent the same elements throughout the drawings and written description, and therefore their description will not be repeated.

[0039] One or more example embodiments of the present disclosure relate to a multi-learning strategy framework for a computer vision training system. The multi-learning strategy framework can combine up to three different machine learning techniques to improve the accuracy of training for a variety of different computer vision applications without additional computational costs for the prediction process. For example, in some embodiments, the multi-learning strategy framework can combine supervised learning techniques, unsupervised learning techniques, and weakly supervised learning techniques to train for a variety of different computer vision applications and scenarios.

[0040] In some embodiments, a supervised output (e.g., a supervised target output or a supervised target term) may be generated (e.g., estimated) according to a supervised learning technique. The supervised output may be generated from one or more input images and may correspond to information estimated from the one or more input images that may be desired by a corresponding computer vision (CV) application. For example, the supervised output may correspond to depth, disparity, high-resolution images, multi-view depth, etc. In some embodiments, a supervised loss between the supervised output and a ground-truth of the supervised output may be calculated according to a supervised learning technique.

[0041] In some embodiments, the supervised output can be used as input for calculating an unsupervised loss according to an unsupervised learning technique, and as input for calculating a weakly supervised loss according to a weakly supervised learning technique. For example, the supervised output can be used to transform (e.g., warp) an input image that can be used to calculate an unsupervised loss according to an unsupervised learning technique, and the supervised output can be compared with a weak label to calculate a weakly supervised loss according to a weakly supervised learning technique. Thus, during the optimization of the unsupervised loss and the weakly supervised loss, the accuracy of the supervised output can be further refined (e.g., further improved).

[0042] In some embodiments, the weak labels used to calculate the weak supervision loss can be pixel-based or patch-based and can be generated from the input image or from the ground truth. Thus, in some embodiments, flexibility can be increased compared to computer vision systems that can only support pixel-based weak labels.

[0043] In some embodiments, the multi-learning strategy framework can be applied to any suitable computer vision-based application with multi-frame / multi-image input. The multi-frame / multi-image input can be generated from the same source (e.g., consecutive frames from a single camera) or from different sources (e.g., images from dual cameras or different cameras with different perspectives or different fields of view).

[0044] In some embodiments, a computer vision system can be trained end-to-end for a target computer vision application by optimizing a supervised loss function, an unsupervised loss function, and a weakly supervised loss function concurrently (e.g., jointly, simultaneously, or simultaneously) with each other. In some embodiments, optimizing the supervised loss function, the unsupervised loss function, and the weakly supervised loss function concurrently with each other can include a situation where the optimization processes of the supervised loss function, the unsupervised loss function, and the weakly supervised loss function can at least partially overlap with each other in time.

[0045] Figure 1 is a block diagram of a computer vision training system according to one or more example embodiments of the present disclosure.

[0046] In brief overview, the computer vision (CV) training system 102 may include a deep learning neural network (DNN) framework that is trained for a variety of different types of computer vision (CV) applications based on a variety of machine learning techniques. For example, in some embodiments, the CV training system 102 may utilize supervised learning techniques (S), unsupervised learning techniques (U), and weakly supervised learning techniques (W) to train for a variety of different CV application scenarios. Some non-limiting examples of CV applications may include monocular depth estimation, stereo matching, image / video enhancement, multi-view depth estimation, etc.

[0047] In some embodiments, the CV training system 102 may generate (e.g., estimate) a supervised output corresponding to a target CV application, and may generate (e.g., minimize) a supervised loss function between the supervised output and the corresponding ground truth of the supervised output according to a supervised learning technique (S). The supervised output may correspond to information (e.g., estimated, calculated, extracted, converted, etc.) obtained from one or more input images that may be desired by the target CV application (e.g., may be the target of the target CV application) (e.g., such as depth, disparity, corresponding high-resolution images, multi-view depth, etc.). In other words, the supervised output obtained from the one or more input images may be the target of the corresponding CV application, and the corresponding CV application is used to process, analyze, classify, transform, deform, etc. the one or more input images according to the corresponding target CV application.

[0048] In some embodiments, the supervised output may be used as input to generate an unsupervised loss function according to an unsupervised learning technique (U) and as input to generate a weakly supervised loss function according to a weakly supervised learning technique (W). For example, in some embodiments, the CV training system 102 may transform or deform one or more of the input images based on the supervised output to generate an unsupervised loss function according to the unsupervised learning technique (U), and may compare weak labels corresponding to one or more input images with the supervised output to generate a weakly supervised loss function according to the weakly supervised learning technique (W). As a result, for example, during the optimization of the unsupervised loss function and the weakly supervised loss function, the accuracy of the supervised output may be further improved.

[0049] For more details, refer to Figure 1In some embodiments, the CV training system 102 may receive multi-frame / multi-view images 104 and ground truth values ​​(e.g., ground truth values, ground truth images, etc.) 106. For example, the multi-frame / multi-view images 104 may be captured from one or more imaging devices (e.g., one or more cameras). In this case, the multi-frame / multi-view images 104 may be from the same imaging device (e.g., from the same camera), or may be from different imaging devices (e.g., from dual cameras, two or more different cameras, etc.). For example, in some embodiments, the multi-frame / multi-view images 104 may include multiple consecutive frames captured from the same camera (e.g., from a single camera), or may include multiple views (e.g., stereoscopic views, different fields of view, etc.) captured from two or more cameras (e.g., dual cameras, two or more different cameras, etc.).

[0050] In some embodiments, the ground truth 126 may include one or more values ​​or images corresponding to a target value or image of a supervisory output. For example, in some embodiments, the CV training system 102 may generate (e.g., estimate) a supervisory output according to (e.g., expected by) a target CV application and may compare the supervisory output to the corresponding ground truth 126 of the supervisory output. For example, in some embodiments, the CV training system 102 may generate (e.g., minimize or reduce) a supervisory loss function based on the supervisory output and the corresponding ground truth.

[0051] In some embodiments, the CV training system 102 may include one or more processing circuits 106, including one or more processors 108 and memory 110. Each of the processors 108 may be a general-purpose processor or a special-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGA), a group of processing components, or other suitable processing components. Each of the processors 108 may be integrated into a single device or distributed across multiple separate systems, servers, or devices (e.g., computers). For example, each of the processors 108 may be an internal processor relative to the CV training system 102, or one or more of the processors 108 may be an external processor (e.g., implemented as part of one or more servers or as a cloud-based computing system). Each of the processors 108 may be configured to execute computer code or instructions stored in the memory 110 and / or received from other computer-readable media (e.g., a CDROM, a network storage device, a remote server, etc.).

[0052] The memory 110 may include one or more devices (e.g., memory cells, storage devices, storage devices, etc.) for storing data and / or computer code used to perform and / or facilitate the various processes described in the present disclosure. The memory 110 may include random access memory (RAM), read-only memory (ROM), hard drive storage, temporary storage, non-volatile memory, flash memory, optical storage, or any other suitable memory for storing software objects and / or computer instructions. The memory 110 may include database components, object code components, script components, and / or any other type of information structure used to support the various activities and information structures described in the present disclosure. In one or more processing circuits 106, the memory 110 may be communicatively connected to one or more processors 108, and the memory 110 may include computer code for performing (e.g., executed by one or more processors 108) one or more processes described herein.

[0053] In some embodiments, the memory 110 may include a supervised learning (S) module (e.g., a supervised learning system, a supervised learning neural network, a supervised learning framework, etc.) 112, an unsupervised learning (U) module (e.g., an unsupervised learning system, an unsupervised learning neural network, an unsupervised learning framework, etc.) 114, a weakly supervised learning (W) module (e.g., a weakly supervised learning system, a weakly supervised learning neural network, a weakly supervised learning framework, etc.) 116, a weak label generator 118, and a joint optimizer 120.

[0054] In various embodiments, the supervised learning module 112, the unsupervised learning module 114, the weakly supervised learning module 116, the weak label generator 118, and the joint optimizer 120 may be implemented within a single device (e.g., a single computer, a single server, a single housing, etc.), or at least some of the supervised learning module 112, the unsupervised learning module 114, the weakly supervised learning module 116, the weak label generator 118, and the joint optimizer 120 may be distributed across multiple devices (e.g., across multiple computers, multiple servers, multiple housings, etc.). In various embodiments, each of the supervised learning module 112, the unsupervised learning module 114, the weakly supervised learning module 116, the weak label generator 118, and the joint optimizer 120 may include any suitable processor (e.g., one or more processors 108), memory (e.g., one or more memory devices of the memory 110), encoder-decoder pairs, logic devices, neural networks (e.g., convolutional neural networks (CNNs), recurrent neural networks (RNNs), etc.), controllers, circuits (e.g., integrated circuits (ICs)), etc.) to support the various functions, processes, activities, etc. described in this disclosure.

[0055] In some embodiments, the supervised learning module 112 can be trained using training examples (e.g., using ground truth) to generate a prediction model (e.g., supervised output) corresponding to the target CV application. For example, the training examples may include labels indicating the ground truth of the labels. In some embodiments, the unsupervised learning module 114 can be trained without the benefit of ground truth. According to one or more example embodiments, the unsupervised learning module 114 can analyze, extract, organize, transform, deform, etc., one or more input images based on the supervised output generated by the supervised learning module 112. In some embodiments, the weakly supervised learning module 116 can be trained with both labeled data and unlabeled data. For example, in some embodiments, the weakly supervised learning module 116 can be trained based on weak labels and supervised output.

[0056] For example, the following will refer to Figure 2 Describing in more detail, in some embodiments, the supervised learning module 112 may generate (e.g., estimate) a supervised output based on the multi-frame / multi-view image 104 according to a target CV application. The supervised learning module 112 may generate (e.g., minimize) a supervised loss function between the supervised output and the corresponding ground truth 126. The unsupervised learning module 114 may generate (e.g., minimize) an unsupervised loss function based on the multi-frame / multi-view image 104 and the supervised output. For example, in some embodiments, the unsupervised learning module 114 may transform or deform the multi-frame / multi-view image 104 according to the supervised output, and the transformed or deformed image may be used to generate (e.g., minimize) the unsupervised loss function.

[0057] The weakly supervised learning module 116 may generate (e.g., may minimize) a weakly supervised loss function based on the supervised output. For example, in some embodiments, the weakly supervised learning module 116 may generate (e.g., may minimize) a weakly supervised loss function between a weak label corresponding to the multi-frame / multi-view image 104, which may be generated by the weak label generator 118, and the supervised output. The weak label generator 118 may generate (e.g., pixel-based or image patch-based) weak labels from the multi-frame / multi-view image 104 or from the ground truth 126 corresponding to the supervised output.

[0058] In some embodiments, the joint optimizer 120 may generate (e.g., may minimize) a joint loss function based on the supervised loss function, the unsupervised loss function, and the weakly supervised loss function to train the CV training system 102 according to the target CV application. In some embodiments, corresponding weights may be used in the joint loss function to balance the importance of each of the supervised loss function, the unsupervised loss function, and the weakly supervised loss function in the joint loss function.

[0059] Figure 2is a diagram of a multi-learning strategy framework of a computer vision training system according to one or more example embodiments of the present disclosure.

[0060] Reference Figure 2 In some embodiments, for multi-frame / multi-view images (e.g., I a ,I b ,I c ), the supervised learning module 112 may generate (e.g., estimate) a supervised output (e.g., O) according to the target CV application (e.g., by O=S(I)). a ,O b ,O c In this case, the supervisory output can be minimized or reduced (e.g., a , O b , O c ) and the corresponding ground truth (e.g., ) between the supervision loss (L S ) function to optimize the network. For example, in some embodiments, the supervised learning module 112 may minimize or reduce the supervised loss function

[0061] For non-limiting example, when the CV application corresponds to monocular depth estimation, the supervised learning module 112 may estimate the depth of multiple frames / multi-view images (eg, I a , I b , I c ) in the depth of the object (e.g., D t ) as the supervisory output (e.g., O a , O b , O c In another non-limiting example, when the CV application corresponds to stereo matching, the supervised learning module 112 may estimate the multi-frame / multi-view image (eg, I a , I b , I c ) of the left image (e.g., I l ) and the right image (e.g., I r ) between them (e.g., D) as the supervision output (e.g., O a , O b , O c In yet another non-limiting example, when the CV application corresponds to image / video enhancement, the supervised learning module 112 may be associated with multiple frames / multi-view images (eg, I a , I b , I c ) of low-resolution input frames (e.g., ) corresponding to the high-resolution frame (e.g., ) is estimated as the supervised output (e.g., O a , O b , O c In yet another non-limiting example, when the CV application corresponds to multi-view depth estimation, the supervised learning module 112 may estimate the depth of multiple frames / multi-view images (eg, I a , I b , I c ) in a plurality of views (e.g., D a , D b ) as the supervisory output (e.g., O a , O b , O c ).

[0062] In some embodiments, the unsupervised learning module 114 may receive multiple frames / multi-view images (eg, a , I b , I c ) and supervisory outputs (e.g., O a , O b , O c ) as input, and can minimize or reduce the unsupervised loss (L U ) function. For example, in some embodiments, the unsupervised learning module 114 may minimize or reduce the unsupervised loss function L U (I a I b I c , O a , O b , O c In some embodiments, the weakly supervised learning module 116 may receive an estimated supervised output (e.g., a , O b , O c ), and can minimize or reduce weak labels (e.g., ) and the supervisory output (e.g., O a , O b , O c ) between the weak supervision loss (L w ) function. For example, in some embodiments, the weakly supervised learning module 116 may minimize or reduce the weakly supervised loss function In some embodiments, during training, the weakly supervised loss function Can be optimized.

[0063] In some embodiments, the weak label generator 118 may be configured to generate a weak label from multiple frames / multi-view images (e.g., a , I b , I c ) or directly from the ground truth (e.g., ) to generate weak labels (e.g., For example, in some embodiments, the weak label generator 118 may generate weak labels (e.g., In this case, weak labels (e.g. ) can be generated according to heuristic rules associated with the auxiliary task. As a non-limiting example, some existing knowledge (e.g., such as the sky is always farther than the tree) can be used to generate weak label depth from the semantic segmentation map.

[0064] In some embodiments, the weak tags generated by the weak tag generator 118 (e.g., ) can be pixel-based or image patch-based. Pixel-based weak labels define the relationship between individual pixels of an image, while image patch-based weak labels define the relationship between different cropped regions of an image, where each region includes multiple pixels. That is, image patch-based weak labels may have image patch pairs corresponding to different cropped regions of one or more input images, and image patch-based weak labels may define relationships corresponding to supervisory outputs between the cropped regions of the image patch pairs. Therefore, image patch-based weak labels may be more flexible than pixel-based ordinal weak labels. For example, in the case of image patch-based weak labels, in some embodiments, the weak label generator 118 may generate image patch-based weak labels as image patch pairs (p, q) and factors (e.g., binary relationships, distributions, statistical information, etc.). During training, the weakly supervised learning module 116 may check the supervisory outputs (e.g., O a , O b , O c ) image patch pair (p, q) and the generated weak label Is it consistent (eg, similar)? For a non-limiting example, in the case of CV applications for image / video enhancement, if p is from the sky and q is from trees, then p should be "bluer" than q in super-resolution.

[0065] In some embodiments, the CV training system 102 may, for example, concurrently optimize the supervision loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function to perform end-to-end training. For example, in some embodiments, the joint optimizer 120 (e.g., see Figure 1 ) can be calculated based on the supervisory loss (L S ) function, unsupervised loss (L U ) function, weak supervision loss (L w ) function and the corresponding weight (w s , wu , w w ) to generate the joint loss function (L all ). Weight (w s , w u , w w ) can be used to balance the joint loss function (L all ) in the supervision loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function. In some embodiments, the weight (w s , w u , w w ) may be determined (e.g., may be set) based on an ablation study for a target CV application. For example, in some embodiments, the joint optimizer 120 may be configured according to the joint loss function (L all ) to generate (e.g., minimize or reduce) the joint loss function (L all ) to train a CV training system 102 for a corresponding CV application.

[0066] Equation 1:

[0067] L all =w s L s +w u L u +w w L w

[0068] In Equation 1, L s It can correspond to the supervised loss function, w s This corresponds to the loss function (L s ) weight, L u It can correspond to the unsupervised loss function, w u This corresponds to applying an unsupervised loss function (L u ) weight, L w It can correspond to the weak supervision loss function, w w It can be applied to the weak supervision loss function (L w )’s weight.

[0069] Although Figure 2 Shows multi-frame / multi-view images (I a , I b , I c) includes three input frames or views, but the present disclosure is not limited thereto. For example, the multi-learning strategy framework of the CV training system 102 can be applied to training for any suitable CV application with any suitable number of frames or views. In addition, although Figure 2 The CV training system 102 is shown to include a supervised learning module 112, an unsupervised learning module 114, and a weakly supervised learning module 116, but the present disclosure is not limited thereto, and the CV training system 102 may use (e.g., may include) any suitable combination of the supervised learning module 112, the unsupervised learning module 114, and the weakly supervised learning module 116. For example, if the ground truth value of the supervised term is not available, the CV training system 102 may use the unsupervised learning module 114 and the weakly supervised learning module 116. As another example, if the multi-frame / multi-view image 104 includes only a single frame / single image input, the CV training system 102 may use the supervised learning module 112 and the weakly supervised learning module 116.

[0070] In some embodiments, the multi-frame / multi-view image (I a , I b , I c ) can be used to train the CV training system 102 (eg, during training and / or development), but the multi-frame / multi-view image (I a , I b , I c ) end-user devices that may not be used for the prediction process of the target CV application are needed (e.g., may not be used or may not be required) in order to utilize the training of the CV training system 102 for the target CV application. For example, the prediction process of the target CV application of the end-user device may be based on a single frame image input, and the prediction time may not increase or substantially increase. Therefore, in addition to video-based CV applications, the CV training system 102 can be used for image-based CV applications with a single frame image input.

[0071] Figure 3 is a diagram of a multi-learning strategy framework of a computer vision training system for monocular depth estimation according to one or more example embodiments of the present disclosure.

[0072] In one or more exemplary embodiments of the present disclosure, the multi-learning strategy framework of the CV training system 102 can be applied to monocular depth estimation. Typically, monocular depth estimation is a CV application that can be used to estimate a pixel-wise depth map of an image.

[0073] In brief overview, given a plurality of consecutive input frames, the supervised learning module may estimate the depth of the object shown in the input frame as a supervised output, and may calculate a supervised loss between the estimated depth and the ground truth of the depth. The unsupervised learning module 114 may use the estimated depth to compensate for the rigid motion of the object in the input frame in the 3D space, and may calculate an unsupervised photometric loss between the input frames based on the 3D coordinates and the rigid motion in the 3D space. The weakly supervised learning module 114 may calculate a weakly supervised loss between the estimated depth and a weakly labeled depth that may be generated from the input frame or from the ground truth. In some embodiments, the CV training system 102 may be trained by concurrently optimizing the supervised loss, the unsupervised photometric loss, and the weakly supervised loss.

[0074] For more details, refer to Figure 3 In some embodiments, the CV training system 102 may receive a plurality of consecutive input image frames (I t-1 , I t , I t+1 For example, the input image frame (I t-1 , I t , I t+1 ) may include a previous frame (t-1), a target frame (t), and a next frame (t+1) which may be adjacent to each other.

[0075] In some embodiments, the supervised learning module 112 may estimate the depth (D t ) as a supervised output. For example, in some embodiments, the supervised learning module 112 may include a depth estimator (e.g., a depth estimation network) 308 to estimate the depth (D) using any suitable depth estimation method or algorithm known to those skilled in the art (e.g., such as conditional random fields, encoder-decoder networks, etc.). t ). For example, in some embodiments, the depth estimator 308 may include one or more encoder-decoder pairs (e.g., a network of encoder-decoder pairs) to estimate the depth (D) of the object in the target frame (t). t ).

[0076] In some embodiments, the supervised learning module 112 may calculate the estimated depth (D t ) and the ground truth of depth The supervision loss between s For example, in some embodiments, the supervised learning module 112 may calculate the supervised loss (L s ).

[0077] Equation 2:

[0078]

[0079] In Equation 2, L s It can correspond to the supervision loss, D t may correspond to the estimated depth output by the depth estimator 308, It can correspond to the estimated depth (D t )’s ground truth.

[0080] In some embodiments, the unsupervised learning module 114 may use the estimated depth (D t ) to compensate the input image frame (I t-1 , I t , I t+1 ). For example, in some embodiments, the unsupervised learning module 114 may include a pose estimator (e.g., a pose estimation network) 302 and a projective and deformable image transformer 304.

[0081] In some embodiments, the pose estimator 302 may determine the input image frame (I t-1 , I t , I t+1 ). For example, in some embodiments, the pose estimator 302 may determine the rigid motion of the object from the target frame (t) to the previous frame (t-1) (e.g., as M t→t-1 ), and the rigid motion of the object from the target frame (t) to the next frame (t+1) (e.g., as M t→t+1 ). Here, M may be a motion vector of the object, and each motion vector M may include a rotation (R) and a translation (T). The pose estimator 302 may use any suitable pose estimation method or algorithm known to those skilled in the art (e.g., such as linear n-point estimation, pose CNN, or other suitable deep neural network) to determine the rigid motion of the object to output the motion vector M. For example, in some embodiments, the pose estimator 302 may include one or more encoder-decoder pairs (e.g., a network of encoder-decoder pairs) to output the motion vector M of the object based on the rigid motion of the object from frame to frame.

[0082] In some embodiments, the pose estimator 302 may have an encoder-decoder architecture that is similar (e.g., the same or substantially the same) to the encoder-decoder architecture of the depth estimator 308. In some embodiments, at least one of the encoder-decoder pairs of the pose estimator 302 may be shared with the encoder-decoder pair of the depth estimator 308. Thus, computational cost may be reduced. However, the present disclosure is not limited in this regard.

[0083] In some embodiments, the projected and deformed image converter 304 may compensate for thet-1 , I t , I t+1 ) in the 2D image and can be used to estimate the depth (D t ) transforms the compensated 2D image into a 3D space (e.g., 3D coordinates). For example, in some embodiments, the projected and deformed image transformer 304 may transform the input image frame (I t-1 , I t , I t+1 ) to transform the input image frame (I t-1 , I t , I t+1 ) is isolated from rigid motion. In some embodiments, the projected and deformed image converter 304 may be configured to project objects in the 2D image according to the estimated depth (D t ) The 2D coordinates of the object in the compensated 2D image are projected (eg, converted) into 3D coordinates. For example, the 2D coordinates may correspond to the image coordinates (x, y) of the object in the compensated 2D image, and the estimated depth (D t ) may provide the depth dimension of the object corresponding to the third coordinate (z) of the 3D coordinate. Thus, in some embodiments, the projected and deformed image transformer 304 may generate an estimated depth (D t ) to compensate for the rigid motion of objects in 3D space (e.g., 3D coordinates).

[0084] In some embodiments, the unsupervised learning module 114 may calculate the input image frame (I t-1 , I t , I t+1 ) and rigid motion compensation between the unsupervised photometric loss (L u For example, in some embodiments, the unsupervised learning module 114 may calculate the unsupervised photometric loss (L u ).

[0085] Equation 3:

[0086] L u =||I t-1 -K.M. t→t-1 °(D t K -1 I t )|| 2 +||I t+1 -K.M. t→t+1 °(D t K -1 I t )||2

[0087] In Equation 3, L u It can correspond to the unsupervised loss, I t-1 It can correspond to the previous input image frame, I t It can correspond to the target input image frame, I t+1 may correspond to the next input image frame, K may correspond to the intrinsic matrix, M t→t-1 It can correspond to the input image frame (I t ) to the previous input image frame (I t-1 ) of the rigid motion, M t→t+1 It can correspond to the input image frame (I t ) to the next input image frame (I t+1 ) of the rigid motion, D t may correspond to the estimated depth output by the supervised learning module 112, “·” may correspond to the coordinate projection between 2D and 3D, and “°” may correspond to rigid motion compensation in 3D space.

[0088] In some embodiments, the weakly supervised learning module 116 may calculate the weak label depth corresponding to the target frame (t) and the estimated depth (D) generated by the supervised learning module 112 based on the target frame (t) t ) between the weak supervision loss (L w For example, in some embodiments, the weak label generator 118 may generate a weak label from an input image frame (I t-1 , I t , I t+1 )(or depth from ground truth ) to generate weak label depth In various embodiments, weak tag depth It can be pixel-based or image patch-based. Therefore, the weak label depth Pixel pairs (i, j) or image block pairs (p, q) may be included, and depth relationships (e.g., closer / farther) between pixels of the pixel pair (i, j) or between regions of the image block pair (p, q) may be defined.

[0089] In some embodiments, when the weak tag depth When the weakly supervised learning module 116 is generated to include the pixel pair (i, j), the weakly supervised learning module 116 may calculate the weakly supervised loss (L w ).

[0090] Equation 4:

[0091]

[0092] in,

[0093] P ij =-r ij (logD t，i -logD t，j )

[0094] In Equation 4, L w It can correspond to the weak supervision loss, D t，i may correspond to the estimated depth of pixel i in frame t, D t，j may correspond to the estimated depth of pixel j in frame t, δ may correspond to a constant to avoid discontinuities, P ij may correspond to a constant pixel pair value for pixel i and pixel j. In this case, if pixel i is farther than pixel j, then r ij = 1, otherwise (e.g., if pixel j is farther than pixel i), then r ij =-1.

[0095] In some embodiments, when the weak tag depth When the image block pair (p, q) is generated, the weakly supervised learning module 116 may calculate the weakly supervised loss (L w ).

[0096] Equation 5:

[0097]

[0098] in,

[0099] P ij =-r ij (log(mean(D t，p ))-log(mean(D t，q )))

[0100] In Equation 5, L w It can correspond to the weak supervision loss, D t，p may correspond to the estimated depth of region p in frame t, D t，q may correspond to the estimated depth of region q in frame t, and var may correspond to the variance to suppress outliers (e.g., when computing P ij In this case, if region p is farther than region q, then r ij = 1, otherwise (for example, if region q is farther than region p), then r ij =-1.

[0101] In some embodiments, when the weak tag depth is generated to include the image patch pair (p, q), at the estimated depth (D t ) can be evaluated. For example, if region p and region q are from the same semantic class, the depth distributions of region p and region q may be similar to each other. In this case, the weakly supervised learning module 116 may estimate the depth of region p based on its estimated depth (D t，p ) and the estimated depth (D t，g ) histograms to calculate the weak supervision loss (L w For example, in some embodiments, the weakly supervised learning module 116 may calculate the weakly supervised loss (L w ).

[0102] Equation 6:

[0103] L w =ChiSquare(hist(D t，p ), (D t，q ))

[0104] In Equation 6, L w It can correspond to weak supervision loss, ChiSquare can correspond to the chi-square distance between two histograms, hist can correspond to histogram operation, D t，p may correspond to the estimated depth of region p in frame t, D t，q may correspond to the estimated depth of region q in frame t.

[0105] In some embodiments, the CV training system 102 can be trained to improve monocular depth estimation by concurrently optimizing a supervised loss function (e.g., as shown in Equation 2), an unsupervised loss function (e.g., as shown in Equation 3), and a weakly supervised loss function (e.g., as shown in Equation 4, Equation 5, or Equation 6). For example, in some embodiments, because the estimated depth (D t ) is used to calculate the supervision loss (L s ), unsupervised loss (L u ) and weak supervision loss (L w ), so the joint optimizer 120 can be based on the joint loss function (L all ) (e.g., as shown in Equation 1) to jointly optimize the supervised loss function, the unsupervised loss function, and the weakly supervised loss function.

[0106] Figure 4 is a diagram of a multi-learning strategy framework of a computer vision training system for stereo matching according to one or more example embodiments of the present disclosure.

[0107] In one or more exemplary embodiments of the present disclosure, the multi-learning strategy framework of the CV training system 102 can be applied to stereo matching (e.g., disparity estimation). Generally, stereo matching (e.g., disparity estimation) is a CV application that can be used to match pixels of images (e.g., left and right images) from different perspectives (e.g., different stereo views) of the same scene.

[0108] In brief overview, given multiple input images (e.g., left and right images) corresponding to different stereo views of the same scene, the supervised learning module 112 may estimate the disparity between the first image (e.g., the left image) and the second image (e.g., the right image) as a supervised output, and may calculate a supervised loss between the estimated disparity and the ground truth of the disparity. The unsupervised learning module 114 may use the estimated disparity to generate a warped image corresponding to one of the first and second images compensated according to the estimated disparity, and may calculate an unsupervised photometric loss between the warped image and a true image (e.g., the original image) of the other of the first and second images (e.g., the image not used to generate the warped image). The weakly supervised learning module 116 may calculate a weakly supervised loss between the estimated disparity and a weakly labeled disparity that may be generated from the corresponding input image in the multiple input images or from the ground truth. In some embodiments, the CV training system 102 may be trained by concurrently optimizing the supervised loss, the unsupervised photometric loss, and the weakly supervised loss.

[0109] For more details, refer to Figure 4 In some embodiments, the CV training system 102 may receive a plurality of input images (I l , I r For example, multiple input images (I l , I r ) may include a left image (I ) which may be generated by different sources (eg, dual cameras or different cameras) such as a left camera and a right camera, for example. l ) and right image (I r ).

[0110] In some embodiments, the supervised learning module 112 may estimate the number of input images (I l , I r ). For example, the disparity may correspond to the left image (I l ) and the right image (I r ), parallax can be similar to rigid motion but in one direction (e.g., the left image (I l ) and the right image (I r) in the shift direction between them). In this case, in some embodiments, the supervised learning module 112 may include a disparity estimator (e.g., a disparity estimation network) 406 to estimate the left image (I ) using any suitable disparity estimation method or algorithm known to those skilled in the art (e.g., such as pixel matching, PSMNet or other suitable deep neural networks, etc.). l ) and the right image (I r ). For example, in some embodiments, the disparity estimator 406 may include a processor trained to estimate the disparity (D) between the left image (I l ) and the right image (I r ) is a neural network (e.g., CNN, RNN, etc.) that calculates the disparity (D) between the two.

[0111] In some embodiments, the supervised learning module 112 may calculate the estimated disparity (D) and the ground truth value of the disparity (D * ) between the supervision loss (L s For example, in some embodiments, the supervised learning module 112 may calculate the supervised loss (L s ).

[0112] Equation 7:

[0113] L s =|DD * |

[0114] In Equation 7, L s may correspond to the supervised loss, D may correspond to the estimated disparity output by the disparity estimator 406, D * may correspond to the ground truth of the estimated disparity (D).

[0115] In some embodiments, the unsupervised learning module 114 may use the estimated disparity (D) to generate a deformed image (I′). In this case, the deformed image (I′) may be obtained from multiple input images (I l , I r For example, in some embodiments, the unsupervised learning module 114 may include a deformation image transformer 402.

[0116] In some embodiments, the warped image converter 402 may transform the left image (I l ) and right image (I r ) to generate a deformed image (I′). For example, the deformed image converter 402 may transform the left image (I l ) and right image (I r ) to generate a deformed image (e.g., I l' or I r '). In this case, since the disparity may correspond to the left image (I l ) and the right image (I r ), so if the estimated disparity (D) is generated correctly, the deformed image (I′) can be compared with another image (e.g., the left image (I l ) and right image (I r ) is the same or substantially the same as the other original image or real image. For example, if the left image (I l ) is used to generate a deformed image (I) according to the estimated disparity (D) l ′), then if the estimated disparity (D) is generated correctly, the deformed image (I l ′) can be compared with the right image (I r ) are the same or substantially the same. Similarly, if the right image (I r ) is used to generate a deformed image (I) according to the estimated disparity (D) r ′), then if the estimated disparity (D) is generated correctly, the deformed image (I r ′) can be compared with the left image (I l ) are the same or substantially the same.

[0117] In some embodiments, the unsupervised learning module 114 may generate a deformed image (I′) based on the estimated disparity (D) and a plurality of input images (I l , I r ) (e.g., the original image that is not used to generate the deformed image (I′)) to calculate the unsupervised photometric loss (L u For example, in some embodiments, the unsupervised learning module 114 may calculate the unsupervised photometric loss (L u ).

[0118] Equation 8:

[0119] L u =||I r -I l ′|| 2 =||I r -W(I l , D)′|| 2

[0120] In Equation 8, L u It can correspond to the unsupervised loss, I l ' may correspond to a deformed image (e.g., corresponding to Figure 4 The left image (I l )), I l may correspond to a deformed image (e.g., Figure 4 The left image in the example shown in l ))’s original image (or real image), I r Can correspond to multiple input images (I l , I r ) in another image (e.g., in Figure 4 The right image in the example shown in r )), D may correspond to the estimated disparity output by the supervised learning module 112, and W may correspond to the deformation operator.

[0121] In some embodiments, the weakly supervised learning module 116 may calculate the weak label disparity The weakly supervised loss (L) between the estimated disparity (D) generated by the supervised learning module 112 w ). Because disparity can be inversely proportional to depth, in some embodiments, weak tag disparity The weak tag generator 118 can be used to generate weak tags with the same depth as the weak tag For example, in some embodiments, the weak label generator 118 may generate the weak label from the original image (or the ground truth disparity (D * )) The information obtained (e.g., semantic segmentation map, etc.) generates weak label disparity In various embodiments, weak label parallax It can be pixel-based or image patch-based. Therefore, weak label disparity Pixel pairs (i, j) or image block pairs (p, q) may be included, and a disparity relationship (e.g., smaller / larger) may be defined, which may be inversely proportional to the depth between the pixels of the pixel pair (i, j) or the areas of the image block pair (p, q) (e.g., if the depth of block p is farther than that of block q, the depth of block p is greater than the depth of block q, and the disparity of block p is less than the disparity of block q).

[0122] In some embodiments, in addition to the estimated disparity (D) and weak label disparity The corresponding information can be used to calculate the weak label disparity The weakly supervised loss (L w ) input, the weakly supervised learning module 116 can use the same Figure 3 The weakly supervised loss function for monocular depth estimation discussed above (e.g., Equation 4, Equation 5, or Equation 6) is used to calculate the weakly supervised loss for stereo matching (L w ). For example, as will be understood by one of ordinary skill in the art, the weak supervision loss (L w ) can be achieved by using parallax with weak labels The estimated disparity information of each corresponding pixel (i, j) or region (p, q) is used instead of the above referenced Figure 3 The monocular depth estimation discussed corresponds to the weak label depth The estimated depth (D t ) is calculated according to any appropriate equation among Equation 4, Equation 5, or Equation 6 discussed above. Therefore, its redundant description may not be repeated.

[0123] In some embodiments, the CV training system 102 can be trained to optimize a supervised loss function (e.g., as shown in Equation 7), an unsupervised loss function (e.g., as shown in Equation 8), and a weakly supervised loss function (e.g., with the estimated disparity (D) and the weakly labeled disparity 4, 5 or 6) to improve stereo matching. For example, in some embodiments, because the estimated disparity (D) can be used to calculate the supervision loss (L s ), unsupervised loss (L u ) and weak supervision loss (L w ), so the joint optimizer 120 can be based on the joint loss function (L all ) (e.g., as shown in Equation 1) to jointly optimize the supervised loss function, the unsupervised loss function, and the weakly supervised loss function.

[0124] Figure 5A and Figure 5B is a diagram of a multi-learning strategy framework of a computer vision training system for image / video enhancement according to one or more example embodiments of the present disclosure.

[0125] In one or more exemplary embodiments of the present disclosure, the multi-learning strategy framework of the CV training system 102 can be applied to image / video enhancement. Generally, image / video enhancement is a CV application that can be used to adjust (e.g., enhance) a low-resolution image to a high-resolution image. Figure 5A and Figure 5B A non-limiting example of image / video enhancement in the context of a super-resolution (SR) network is described, but the present disclosure is not limited thereto. As will be understood by one of ordinary skill in the art, reference is made to Figure 5A and Figure 5B The same or substantially the same (eg, similar) architecture described (eg, the multiple learning strategy framework) can be used for other image / video enhancement CV applications with appropriate modifications.

[0126] In brief overview, given a plurality of consecutive low-resolution (LR) input frames, the supervised learning module 112 may estimate a corresponding high-resolution (HR) image as a supervised output, and may calculate a supervised loss between the estimated HR image and the ground truth of the HR image. The unsupervised learning module 114 may generate (e.g., may estimate) an optical flow corresponding to the estimated HR image, and may compensate for optical motion in the estimated HR image based on the optical flow. The unsupervised learning module 114 may calculate an unsupervised photometric loss between the deformed estimated HR image. The weakly supervised learning module 114 may calculate a weakly supervised loss between the estimated HR image and a weakly labeled HR image that may be generated from the LR input frames or from the ground truth. In some embodiments, the CV training system 102 may be trained by concurrently optimizing the supervised loss, the unsupervised photometric loss, and the weakly supervised loss.

[0127] For more details, refer to Figure 5A and Figure 5B In some embodiments, the CV training system 102 may receive a plurality of consecutive LR input image frames that may be generated by a single source (e.g., a single camera). For example, the LR input image frame A first frame (t-1), a second frame (t), and a third frame (t+1) that may be adjacent to each other may be included.

[0128] In some embodiments, the supervised learning module 112 may input frames from the LR Estimating HR image As a supervised output. For example, in some embodiments, the supervised learning module 112 may include a single image super-resolution (SISR) network 510 to use any suitable method or algorithm known to those skilled in the art (e.g., such as upsampling (e.g., bilinear / bicubic upsampling), dictionary learning based on sparse codes, CT-SRCNN or other suitable neural networks, etc.) to obtain the image super-resolution from the LR input frame. Estimating HR image For example, in some embodiments, the SISR network 510 may include a network trained to receive frames from the LR input. Estimating HR image Neural networks (e.g., CNN, RNN, etc.).

[0129] In some embodiments, the supervised learning module 112 may calculate the estimated HR image and the ground truth of the HR image (or called the ground truth HR image) The supervision loss between s ). For example, in some embodiments, the supervised learning module 112 may calculate the estimated HR image according to the supervised loss function shown in Equation 9 and the ground truth HR image The supervision loss (L s ).

[0130] Equation 9:

[0131] L s =||I HR -I HR* || 2

[0132] In Equation 9, I HR may correspond to the estimated HR image in the corresponding frame pair Any one of HR* may correspond to the corresponding ground truth HR images in the corresponding frame pair

[0133] In some embodiments, the unsupervised learning module 114 may compensate the estimated HR image For example, in some embodiments, the unsupervised learning module 114 may generate (eg, may estimate) HR optical flow To compensate the estimated HR image Optical motion in . Optical motion can be similar to rigid motion, but can be global rather than object-based. Thus, while rigid motion considers 3D motion (e.g., including the depth dimension), optical motion can consider the estimated HR image Movement in 2D space (2D image coordinates).

[0134] In various embodiments, the unsupervised learning module 114 may be, for example, Figure 5A The LR input frame is generated (eg, estimated) as shown in HR optical flow from frame to frame Or you can, for example, Figure 5B As shown in , the estimated HR image is directly generated (eg, can be estimated) HR optical flow from frame to frame Therefore, in some embodiments, the unsupervised learning module 114 may be as follows Figure 5A The optical flow estimator 502 and the optical flow SR generator 504 are shown in FIG. Generate HR optical flow Or you can Figure 5B The optical flow estimator 503 is shown in FIG. Directly generate HR optical flow

[0135] Reference Figure 5A In some embodiments, the optical flow estimator 502 may generate an LR input frame LR optical flow from frame to frame For example, in some embodiments, the optical flow estimator 502 may generate an LR input frame LR optical flow from frame (t) to frame (t-1) And the LR optical flow from frame (t) to frame (t+1) The optical flow estimator 502 may use any suitable optical flow estimation method or algorithm known to those skilled in the art (e.g., such as TV-L1 optical flow estimation, PWCNet or other suitable neural networks, etc.) to generate the LR optical flow. For example, in some embodiments, the optical flow estimator 502 may include an optical flow estimator trained to estimate the LR input frame LR optical flow from frame to frame Neural networks (e.g., CNN, RNN, etc.).

[0136] In some embodiments, the optical flow SR generator 504 may generate the LR optical flow Upsampling is performed to generate (e.g., estimate) HR optical flow For example, in some embodiments, the optical flow SR generator 504 may use any suitable upsampling method or algorithm (e.g., such as bilinear / bicubic interpolation) to upsample the LR optical flow. Upsampling may be performed, or may include being trained to extract optical flow from LR Generate HR optical flow Deep neural networks (e.g., CNN, RNN, etc.)

[0137] refer to Figure 5B In some embodiments, the optical flow estimator 503 may directly generate an estimated HR image HR optical flow from frame to frame In this case, the optical flow SR generator 504 (see, for example, Figure 5A ) may be omitted. For example, in some embodiments, the optical flow estimator 503 may be configured to estimate the HR image Directly generate HR optical flow from frame (t) to frame (t-1) and the HR optical flow from frame (t) to frame (t+1) The optical flow estimator 503 may use any suitable optical flow estimation method or algorithm known to those skilled in the art (e.g., such as TV-L1 optical flow estimation, PWCNet or other suitable neural networks, etc.) to generate the HR optical flow. For example, in some embodiments, the optical flow estimator 503 may include an HR image trained to directly estimate the estimated HR optical flow from frame to frame Neural networks (e.g., CNN, RNN, etc.).

[0138] In some embodiments, the unsupervised learning module 114 may use HR optical flow (e.g., output by the optical flow SR generator 504 or by the optical flow estimator 503) and the estimated HR image To compensate the estimated HR image For example, Figure 5A and Figure 5B As shown, in some embodiments, the unsupervised learning module 114 may include a 2D deformable image transformer 506. In some embodiments, the 2D deformable image transformer 506 may be based on the HR optical flow The estimated HR image 2D image deformation to combine optical motion with the estimated HR image Separation.

[0139] In some embodiments, the unsupervised learning module 114 may be based on HR optical flow and the estimated HR image To calculate the unsupervised photometric loss (L u For example, in some embodiments, the unsupervised learning module 114 may calculate the unsupervised photometric loss (L u ).

[0140] Equation 10:

[0141]

[0142] In Equation 10, L u can correspond to the unsupervised loss, May correspond to the first LR input image frame The estimated high-resolution image of May correspond to the second LR input image frame The estimated high-resolution image of May correspond to the third LR input image frame The estimated high-resolution image of can correspond to the optical flow from frame t to frame t-1, may correspond to the optical flow from frame t to frame t+1, “°” may correspond to the 2D deformation, and W may correspond to the deformation operator.

[0143] In some embodiments, the weakly supervised learning module 116 may calculate the weakly labeled HR frames and the corresponding estimated HR image generated by the supervised learning module 112 The weak supervision loss (L wFor example, in some embodiments, the weak label generator 118 may input an image frame from the LR The obtained information (e.g., semantic segmentation map, etc.) generates weakly labeled LR frames In this case, in some embodiments, the weakly supervised learning module 116 may include a method for Upsampling is performed to generate weakly labeled HR frames For example, in other embodiments, the weak label generator 118 may be configured to generate a weak label from a ground truth HR image. Directly generate weakly labeled HR frames And in this case, the upsampler 514 can be omitted.

[0144] In various embodiments, weakly tagged HR frames It can be pixel-based or image patch-based. Therefore, weakly labeled HR frames A pixel pair (i, j) or an image patch pair (p, q) may be included, and an intensity relationship (eg, brighter / darker) between pixels of the pixel pair (i, j) or regions of the image patch pair (p, q) may be defined.

[0145] In some embodiments, in addition to the estimated HR image and weakly labeled HR frames The corresponding information can be used to calculate the weak label HR frame With the estimated HR image The weak supervision loss (L w ) input, the weakly supervised learning module 116 can use the same Figure 3 The weakly supervised loss function for image / video enhancement (L w ). For example, as will be understood by one of ordinary skill in the art, the weak supervision loss (L w ) can be achieved by using the HR frame corresponding to the weak label The estimated HR image of each corresponding pixel (i, j) or region (p, q) intensity relationship (e.g., brighter / darker) rather than using the above reference Figure 3 The monocular depth estimation discussed corresponds to the weak label depth The estimated depth (D) of each corresponding pixel (i, j) or region (p, q) t ) is calculated according to any appropriate equation among Equation 4, Equation 5, or Equation 6 discussed above. Therefore, a redundant description thereof may not be repeated.

[0146] In some embodiments, the CV training system 102 can be trained to obtain the best performance by optimizing a supervised loss function (e.g., as shown in Equation 9), an unsupervised loss function (e.g., as shown in Equation 10), and a weakly supervised loss function (e.g., with the estimated HR image and weakly labeled HR frames 4, 5 or 6) to improve image / video enhancement. For example, in some embodiments, because the estimated HR image Can be used to calculate the supervision loss (L s ), unsupervised loss (L u ) and weak supervision loss (L w ), so the joint optimizer 120 can be based on the joint loss function (L all ) (e.g., as shown in Equation 1) to jointly optimize the supervised loss function, the unsupervised loss function, and the weakly supervised loss function.

[0147] Figure 6 is a diagram of a multi-learning strategy framework of a computer vision training system for multi-view depth estimation according to one or more example embodiments of the present disclosure.

[0148] In one or more example embodiments of the present disclosure, the multi-learning strategy framework of the CV training system 102 can be applied to multi-view depth estimation. In general, multi-view depth estimation is a CV application that can be used to align pixel-by-pixel depth maps of different input images with different views (e.g., different fields of view) with each other. Similar to stereo matching, the input images for multi-view depth estimation can be generated from different sources (e.g., different cameras), but unlike stereo matching where different sources can be well aligned with each other, the sources (e.g., cameras) that generate the multiple views for multi-view depth estimation may not be well aligned. For non-limiting examples, one view can be a front view of an object or scene, and the other view can be a side view of the object or scene.

[0149] In brief overview, given a plurality of input images comprising different views from one another (e.g., different fields of view), the supervised learning module may estimate the depth of an object shown in each input image as a supervised output, and may calculate a supervised loss between the estimated depth of each input image and the ground truth corresponding to the estimated depth of each input image. The unsupervised learning module 114 may generate a 3D point cloud for each input image based on the estimated depth of each input image, and may calculate an unsupervised photometric loss based on the distance between the 3D coordinates of aligned feature points in the 3D point cloud of the input image. The weakly supervised learning module 114 may calculate a weakly supervised loss between the estimated depth of each input image and a weakly labeled depth for each input image that may be generated from the input image or from the ground truth. In some embodiments, the CV training system 102 may be trained by concurrently optimizing the supervised loss, the unsupervised photometric loss, and the weakly supervised loss.

[0150] For more details, refer to Figure 6 In some embodiments, the CV training system 102 may receive a plurality of input images (I a ,I b For example, multiple input images (I a ,I b ) may include a first image (I ) which may be generated by different sources (e.g., different cameras) having different fields of view from each other (e.g., such as a front camera and a side camera). a ) and the second image (I b ). Although as a simplified example, Figure 6 Showing two images (I a ) and (I b ) as the input image (I a , I b ), but the present disclosure is not limited thereto, and the input image may include more than two input images (I a , I b ).

[0151] In some embodiments, the supervised learning module 112 may estimate the a , I b ) in the depth (D a , D b ) as a supervised output. For example, in some embodiments, the supervised learning module 112 may include a depth estimator (e.g., a depth estimation network) 606 to estimate each input image (I a , I b) in the depth (D a , D b For example, in some embodiments, the depth estimator 606 may include one or more encoder-decoder pairs (e.g., a network of encoder-decoder pairs) to estimate the depth of each input image (I a , I b ) in the depth (D a , D b ).

[0152] In some embodiments, the supervised learning module 112 may calculate the a , I b ) of the estimated depth (D a , D b ) and each input image (I a , I b ) is the ground truth of the estimated depth The supervision loss between s For example, in some embodiments, the supervised learning module 112 may calculate the supervised loss (L s ).

[0153] Equation 11:

[0154]

[0155] In Equation 11, L s It can correspond to the supervision loss, D v may correspond to the estimated depth (D a , D b ), It can correspond to the estimated depth (D a , D b ) in the corresponding ground truth value

[0156] In some embodiments, the unsupervised learning module 114 may include a back-projection and calibration image transformer 602 to generate a back-projection image according to the input image (I a , I b ) of the estimated depth (D a , D b ) Each input image (I a , I b ) is inversely projected into 3D space (e.g., camera coordinates), and further inversely projected into world coordinates via a non-intrinsic matrix. In some embodiments, the inverse projection and calibration image transformer 602 may generate each input image (I a , I b) in world coordinates, and may calibrate (eg, may align) common feature points corresponding to each other in each 3D point cloud. In other words, in some embodiments, the inverse projection and calibration image transformer 602 may be based on the estimated depth (D a , D b ) Identify the input image (I) according to the 3D world coordinates a , I b ) in the common area, and the input image (I a , I b )'s 3D world coordinates are calibrated (eg, alignable) with each other.

[0157] In some embodiments, the unsupervised learning module 114 may be configured to generate a a , I b ) is used to calculate the unsupervised photometric loss (L u For example, in some embodiments, the unsupervised learning module 114 may calculate the unsupervised photometric loss (L u ).

[0158] Equation 12:

[0159]

[0160] In Equation 12, L u may correspond to the unsupervised loss, x may correspond to pixels that appear in the common area of ​​the two cameras, and I a (x) may correspond to the 3D world coordinate of pixel x aligned by the first camera, I b (x) may correspond to the 3D world coordinate of pixel x aligned by the second camera, ||.|| 2 It can correspond to the Euclidean distance between two coordinates.

[0161] In some embodiments, the weakly supervised learning module 116 may calculate the a , I b ) corresponding to the weak label depth and each input image (I a , I b ) of the estimated depth (D a , D b ) between the weak supervision loss (L w For example, in some embodiments, the weak label generator 118 may generate a a , I b )'s weak tag depth And the weakly supervised learning module 116 can be used in the same way as above Figure 3 Weak label depth is calculated in the same or substantially the same way as described in the case of monocular depth estimation With the estimated depth (D a , D b ) between the weak supervision loss (L w ), therefore, its redundant description may not be repeated.

[0162] In some embodiments, the CV training system 102 can be trained to improve multi-view depth estimation by concurrently optimizing a supervised loss function (e.g., as shown in Equation 11), an unsupervised loss function (e.g., as shown in Equation 12), and a weakly supervised loss function (e.g., as shown in Equation 4, Equation 5, or Equation 6). For example, in some embodiments, because the estimated depth (D a , D b ) can be used to calculate the supervision loss (L s ), unsupervised loss (L u ) and weak supervision loss (L w ), so the joint optimizer 120 can be based on the joint loss function (L all ) (e.g., as shown in Equation 1) to jointly optimize the supervised loss function, the unsupervised loss function, and the weakly supervised loss function.

[0163] Table 1 below shows the effectiveness of various different multi-learning strategy frameworks of the CV training system 102 according to one or more example embodiments of the present disclosure. As a non-limiting example, Table 1 shows the effectiveness of using the KITTI street view dataset as a training dataset, as applied to the above reference Figure 3 Various learning strategy frameworks for the CV training system 102 for monocular depth estimation CV applications are described.

[0164] Table 1:

[0165]

[0166]

[0167] In Table 1, RMSE may correspond to root mean square error, REL may correspond to relative error, S may represent only a supervised learning framework, S+U may represent a supervised learning framework plus an unsupervised learning framework, S+W may represent a supervised learning framework plus a weakly supervised learning framework, and S+U+W may represent a supervised learning framework plus an unsupervised learning framework plus a weakly supervised learning framework.

[0168] As shown in Table 1, each of the supervised learning framework plus the unsupervised learning framework S+U and the supervised learning framework plus the weakly supervised learning framework S+W performs better than the supervised learning framework S alone. In addition, the supervised learning framework plus the unsupervised learning framework plus the weakly supervised learning framework S+U+W can perform better than each of the supervised learning framework plus the unsupervised learning framework S+U and the supervised learning framework plus the weakly supervised learning framework S+W. Therefore, as described above according to one or more exemplary embodiments of the present disclosure, a more accurate deep learning neural network can be provided by combining at least two of these frameworks.

[0169] Figure 7A and Figure 7B is a table showing an overview of a multi-learning strategy framework of a computer vision training system applied to various computer vision applications described above according to one or more example embodiments of the present disclosure. Figure 8 is a flowchart of a method for training a computer vision training system according to one or more example embodiments of the present disclosure. Figure 8 The order or number of operations of the method 800 shown in FIG. 8 is not intended to be construed as a single operation, and may be changed to any desired order or number of operations as recognized by one of ordinary skill in the art. For example, in some embodiments, the order may be varied, or the method may include fewer or additional operations. Furthermore, the operations shown in the method 800 may be performed by any suitable one of the components of one or more example embodiments described above, or by any suitable combination of components.

[0170] Reference Figure 7A 、 Figure 7B and Figure 8 , a multi-frame / multi-image input may be received by the CV training system 102 and the method 800 may begin. For example, depending on the target CV application, the multi-frame / multi-image input may be generated by a single source (e.g., a single camera) or by multiple sources (e.g., dual cameras or different cameras). At block 805, a supervision output of the multi-frame / multi-image input may be estimated. In some embodiments, the supervised learning module 112 may estimate the supervision output according to the target CV application. For example, if the target CV application corresponds to monocular depth estimation, the supervision output (e.g., the estimated supervision term) may correspond to depth (D t ), if the target CV application corresponds to stereo matching, the supervision output (e.g., the estimated supervision term) may correspond to disparity (D), and if the target CV application corresponds to image / video enhancement, the supervision output (e.g., the estimated supervision term) may correspond to HR image And / or if the target CV application corresponds to multi-view depth estimation, the supervision output (e.g., the estimated supervision term) may correspond to the multi-view depth (D a , D b ).

[0171] In some embodiments, at block 810, a supervision loss may be determined based on the estimated supervision output and the ground truth. For example, in some embodiments, the supervised learning module 112 may determine (e.g., may generate, may calculate, may minimize, etc.) a supervision loss (L s ) function. In this case, for example, if the target CV application corresponds to monocular depth estimation, the supervised loss (L s ) can be used at depth (D t ) and the ground truth depth If the target CV application corresponds to stereo matching, the supervision loss (L s ) can be found in the disparity (D) and the ground truth disparity (D * ), if the target CV application corresponds to image / video enhancement, then the supervision loss (L s ) can be used in HR images Compared with the ground truth HR image , and / or if the target CV application corresponds to multi-view depth estimation, the supervision loss (L s ) can be used in multi-view depth (D a , D b ) and ground truth multi-view depth Corresponding between.

[0172] In some embodiments, at block 815, an unsupervised loss may be determined based on the multi-frame / multi-image input and the estimated supervised output. For example, in some embodiments, the unsupervised learning module 114 may use the supervised output to transform or deform an image corresponding to one or more of the multi-frame / multi-image input. For example, if the target CV application corresponds to monocular depth estimation, the transformed or deformed image may correspond to rigid motion compensation and depth estimation based on the estimated depth (D t ), if the target CV application corresponds to stereo matching, the transformed or deformed image may correspond to the deformed image (I′) generated based on the estimated disparity (D), if the target CV application corresponds to image / video enhancement, the transformed or deformed image may correspond to optical flow estimation and HR image based on the estimation , and / or if the target CV application corresponds to multi-view depth estimation, the transformed or deformed image may correspond to a perspective projection and an estimated multi-view depth (D a , D b ) in a 3D point cloud.

[0173] In some embodiments, the unsupervised learning module 114 may use the transformed or deformed image to determine (eg, generate, calculate, minimize, etc.) an unsupervised loss (Ls ) function. For example, in some embodiments, if the target CV application corresponds to monocular depth estimation, the unsupervised loss (L u ) function can correspond to the rigid motion compensation and the depth based on the estimation (D t ) and the photometric loss between the 2D to 3D coordinate projections of the corresponding original (or real image) of the multi-frame / multi-image input. If the target CV application corresponds to stereo matching, the unsupervised loss (L u ) function may correspond to the photometric loss between the generated deformed image (I′) based on the estimated disparity (D) and the original (or true image) of the deformed image, or the unsupervised loss (L) if the target CV application corresponds to image / video enhancement. u ) function can correspond to the optical flow estimation and the HR image based on the estimation and the photometric loss between the 2D HR image deformations of the estimated HR optical flow, and / or the unsupervised loss (L u ) function may correspond to the multi-view depth (D a , D b )’s distance between aligned points in the 3D point cloud.

[0174] In some embodiments, at block 820, weak labels (e.g., weakly supervised labels) corresponding to the multi-frame / multi-image input may be generated. For example, in some embodiments, the weak label generator 118 may generate weak labels from the multi-frame / multi-image input or from corresponding ground truth values. The weak labels generated by the weak label generator 118 may be pixel-based or image patch-based, and may define suitable relationships between pairs of pixels or image patches for the target CV application (e.g., depth relationships, disparity relationships, intensity relationships, etc.).

[0175] In some embodiments, at block 825, a weak supervision loss may be determined based on the weak labels and the estimated supervision output. For example, in some embodiments, the weak supervision learning module 116 may determine (e.g., may generate, may calculate, may minimize, etc.) a weak supervision loss (L w ) function. In this case, if the target CV application corresponds to monocular depth estimation, the weakly supervised loss (L w ) function can be used to evaluate the weak label depth With the estimated depth (D t ), if the target CV application corresponds to stereo matching, then the weak supervision loss (L w ) function can be used to evaluate weak label disparity and the consistency between the estimated disparity (D), if the target CV application corresponds to image / video enhancement, then the weakly supervised loss (Lw ) function can be used to evaluate weakly labeled HR frames With the estimated HR image and / or if the target CV application corresponds to multi-view depth estimation, the weakly supervised loss (L w ) function can be used to evaluate the weak label depth With the estimated depth (D a , D b ) between them.

[0176] In some embodiments, at block 830, a combined loss function may be calculated based on the supervised loss, the unsupervised loss, and the weakly supervised loss. For example, in some embodiments, the joint optimizer 120 may calculate a combined loss function based on the supervised loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function generates (e.g., determinable, computable, minimizable, etc.) a joint loss function (L all ) (e.g., as shown in Equation 1). In some embodiments, the joint optimizer 120 may apply appropriate weights (w s , w u , w w ) to balance the joint loss function (L all ) in the supervision loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function. For example, in some embodiments, the weight (w s , w u , w w ) can be determined (e.g., can be set) based on ablation studies for the corresponding CV application.

[0177] In some embodiments, at block 835, the combined loss function may be optimized and the method 800 may end. For example, in some embodiments, the CV training system 102 may optimize (e.g., minimize) the combined loss function and the supervised loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function corresponding to the joint loss function (L all ) to perform end-to-end training for the corresponding CV application. According to one or more exemplary embodiments of the present disclosure, since the estimated supervision output can be used to calculate the supervision loss (L s ), unsupervised loss (L u ) and weak supervision loss (L w ), so the joint optimizer 120 can use the joint loss function (L all) (e.g., as shown in Equation 1) and concurrently optimize the supervision loss (L S ) function, unsupervised loss (L U ) function and weakly supervised loss (L w ) function. Therefore, the performance of the CV training system 102 can be improved.

[0178] In the accompanying drawings, for clarity, the relative sizes of elements, layers and regions may be exaggerated and / or simplified. For ease of explanation, spatial relative terms such as "under...", "below...", "below...", "below...", "above...", "above..." etc. may be used to describe the relationship between an element or feature and another element or feature as shown in the figure. It will be understood that spatial relative terms are intended to include different directions of the device in use or in operation in addition to the directions depicted in the drawings. For example, if the device in the drawings is flipped, the element described as "under" or "below" or "below" other elements or features will be located "above" other elements or features. Therefore, the example terms "below..." and "below..." may include two directions of "above..." and "below...". The device can be positioned otherwise (e.g., rotated 90 degrees or in other directions), and the spatial relative descriptors used here should be interpreted accordingly.

[0179] It will be understood that although the terms "first," "second," "third," etc. may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or part from another element, component, region, layer, or part. Thus, a first element, first component, first region, first layer, or first part described below may be referred to as a second element, second component, second region, second layer, or second part without departing from the spirit and scope of the present disclosure.

[0180] It will be understood that when an element or layer is referred to as being “on,” “connected to,” or “coupled to” another element or layer, it can be directly “on,” “connected to,” or “coupled to” the other element or layer, or one or more intervening elements or layers may be present. Additionally, it will be understood that when an element or layer is referred to as being “between” two elements or layers, it can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.

[0181] The terms used herein are used to describe the purpose of specific embodiments and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that when the terms "comprises," "comprising," and "having" and variations thereof are used in this specification, the descriptions indicate the presence of narrated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. When a statement such as "at least one of..." follows a column of elements, the statement modifies the entire column of elements rather than the individual elements in the column.

[0182] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation rather than as terms of degree, and are intended to account for the inherent deviations in measured or calculated values ​​that would be recognized by one of ordinary skill in the art. Additionally, when describing embodiments of the present disclosure, the use of "may" means "one or more embodiments of the present disclosure." As used herein, the terms "use," "using," and "being used" may be considered synonymous with the terms "utilizing," "being utilized," and "being utilized," respectively. Additionally, the term "exemplary" is intended to mean an example or illustration.

[0183] The electronic or electrical devices and / or any other related devices or components according to the embodiments of the present disclosure described herein can be implemented using any suitable hardware, firmware (e.g., application specific integrated circuits), software, or a combination of software, firmware, and hardware. For example, the various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. In addition, the various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate. In addition, the various components of these devices can be processes or threads running on one or more processors in one or more computing devices, executing computer program instructions and interacting with other system components for performing the various functions described herein. The computer program instructions are stored in a memory, wherein a standard memory device (such as, for example, a random access memory (RAM)) can be used to implement the memory in the computing device. The computer program instructions can also be stored in other non-temporary computer-readable media (such as, for example, a CD-ROM, a flash drive, etc.). In addition, those skilled in the art will recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present disclosure.

[0184] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will also be understood that, unless expressly defined otherwise herein, terms (such as those defined in common dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense.

[0185] Although some example embodiments have been described, those skilled in the art will readily appreciate that various modifications are possible in the example embodiments without departing from the spirit and scope of the present disclosure. It will be understood that, unless otherwise described, the description of features or aspects within each embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Therefore, it will be understood that the foregoing is an illustration of various example embodiments and should not be construed as limiting the specific example embodiments disclosed herein, and that various modifications to the disclosed example embodiments and other example embodiments are intended to be included within the spirit and scope of the present disclosure as defined in the appended claims and their equivalents.

Claims

1. A computer vision training system comprising: A supervised learning system configured to: estimate a supervision output from one or more input images according to a target computer vision application, and determine a supervision loss based on the supervision output and a ground truth of the supervision output; an unsupervised learning system configured to: determine an unsupervised loss based on a supervised output and the one or more input images, wherein the unsupervised learning system deforms or transforms the one or more input images based on the supervised output and uses the transformed or deformed images to generate the unsupervised loss; A weakly supervised learning system configured to: determine a weakly supervised loss based on the supervised output and the weak labels corresponding to the one or more input images; and The joint optimizer is configured to optimize supervised loss, unsupervised loss, and weakly supervised loss.

2. The system according to claim 1, wherein: The target computer vision application corresponds to monocular depth estimation, and the supervised output corresponds to a depth estimated from a target image among the one or more input images.

3. The system according to claim 2, wherein: The unsupervised learning system is configured to convert 2D image coordinates corresponding to the one or more input images into 3D coordinates according to the estimated depth, and calculate an unsupervised loss based on at least the 3D coordinates.

4. The system according to claim 1, wherein: The target computer vision application corresponds to stereo matching, and the supervised output corresponds to an estimated disparity between a left image and a right image among the one or more input images.

5. The system according to claim 4, wherein: The unsupervised learning system is configured to shift one of the left image and the right image according to the estimated disparity to generate a deformed image, and calculate an unsupervised loss based on at least the deformed image.

6. The system according to claim 1, wherein: The target computer vision application corresponds to image / video enhancement, and the supervised output corresponds to a high-resolution image estimated from a low-resolution input image among the one or more input images.

7. The system according to claim 6, wherein: The unsupervised learning system is configured to estimate an optical flow corresponding to the estimated high-resolution image and to compute an unsupervised loss based on at least the estimated optical flow and the estimated high-resolution image.

8. The system according to claim 1, wherein: The target computer vision application corresponds to multi-view depth estimation, and the supervision output corresponds to an estimated depth in each of a plurality of input frames among the one or more input images.

9. The system according to any one of claims 1 to 8, wherein: The joint optimizer is configured to concurrently optimize the supervised loss, the unsupervised loss, and the weakly supervised loss according to a joint loss function, wherein the joint loss function includes corresponding weights of the supervised loss, the unsupervised loss, and the weakly supervised loss.

10. The system according to any one of claims 1 to 8, further comprising: A weak label generator is used to generate weak labels as image block-based weak labels, wherein the image block-based weak labels have image block pairs corresponding to different cropped areas of the one or more input images, and the image block-based weak labels define a relationship corresponding to the supervisory output between the cropped areas of the image block pairs.

11. A method for training a computer vision system, the method comprising: estimating a supervised output from one or more input images according to a target computer vision application; Determine the supervision loss based on the supervision output and the ground truth of the supervision output; determining an unsupervised loss based on the supervised output and the one or more input images; determining a weak supervision loss based on the supervision output and the weak labels corresponding to the one or more input images; as well as Optimize supervised loss, unsupervised loss and weakly supervised loss, Wherein, the step of determining the unsupervised loss includes: deforming or transforming the one or more input images according to the supervised output, and using the transformed or deformed images to generate the unsupervised loss.

12. The method according to claim 11, wherein The target computer vision application corresponds to monocular depth estimation, and the supervised output corresponds to a depth estimated from a target image among the one or more input images.

13. The method according to claim 12, further comprising: converting 2D image coordinates corresponding to the one or more input images into 3D coordinates according to the estimated depth; as well as Calculate unsupervised loss based on at least 3D coordinates.

14. The method according to claim 11, wherein The target computer vision application corresponds to stereo matching, and the supervised output corresponds to an estimated disparity between a left image and a right image among the one or more input images.

15. The method according to claim 14, further comprising: shifting one of the left image and the right image according to the estimated disparity to generate a deformed image; as well as At least the unsupervised loss is computed based on the deformed image.

16. The method according to claim 11, wherein The target computer vision application corresponds to image / video enhancement, and the supervised output corresponds to a high-resolution image estimated from a low-resolution input image among the one or more input images.

17. The method according to claim 16, further comprising: estimating the optical flow corresponding to the estimated high-resolution image; as well as An unsupervised loss is computed based on at least the estimated optical flow and the estimated high-resolution image.

18. The method according to claim 11, wherein The target computer vision application corresponds to multi-view depth estimation, and the supervision output corresponds to an estimated depth in each of a plurality of input frames among the one or more input images.

19. The method according to any one of claims 11 to 18, wherein: The optimization steps include: The supervised loss, unsupervised loss and weakly supervised loss are optimized concurrently according to a joint loss function, which includes the corresponding weights of the supervised loss, unsupervised loss and weakly supervised loss.

20. The method according to any one of claims 11 to 18, further comprising: The weak labels are generated as patch-based weak labels, the patch-based weak labels having pairs of image patches corresponding to different cropped regions of the one or more input images, the patch-based weak labels defining a relationship corresponding to a supervisory output between the cropped regions of the patch pairs.

Images