Generation of machine learning predictions using multiple domain data sets

Abstract

A method includes obtaining an input matrix and determining a domain index matrix including, for each respective input value of the input matrix, a corresponding domain index value indicating a corresponding training data distribution of a plurality of training data distributions. The method also includes providing the input matrix and the domain index matrix to a machine learning model that has been trained using the plurality of training data distributions, where each respective training data distribution is associated with a different attribute. The method further includes generating, by the machine learning model and based on the input matrix and the domain index matrix, an output matrix including, for each respective input value, a corresponding output value generated based on (i) the respective input value and (ii) the corresponding domain index value, such that the corresponding output value exhibits an attribute of the corresponding training data distribution.

Description

Background Technology

[0001] Machine learning models can be used to process various types of data, including images, time series, text, and / or point clouds, etc. Improvements in machine learning models allow them to perform data processing faster and / or with less computational resources. Summary of the Invention

[0002] Multiple training data distributions can be used to train a machine learning model to perform a specific task. These distributions can be distinct from each other, each associated with different attributes. The machine learning model can be configured to generate an output matrix based on an input matrix and a domain index matrix. The input matrix represents the input data that will be processed by the machine learning model as part of the performance of the specific task. The domain index matrix controls the degree to which each of the multiple training data distributions influences the output matrix. Specifically, the domain index value of each element of the domain index matrix controls an attribute of one or more corresponding output values ​​of the output matrix. Therefore, the attributes of the output matrix can be changed by altering the domain index values ​​of the domain index matrix, with the possible range of attributes based on the multiple training data distributions.

[0003] In a first example embodiment, a method may include obtaining an input matrix representing input data. The method may further include determining a domain index matrix comprising a corresponding domain index value for each corresponding input value of the input matrix, the corresponding domain index value indicating a corresponding training data distribution to be used in generating an output value corresponding to the corresponding input value. The corresponding training data distribution may be one of multiple training data distributions. The method may additionally include providing an input matrix and a domain index matrix to a machine learning model trained to generate an output matrix using multiple training data distributions. Each corresponding training data distribution of the multiple training data distributions may be associated with a property of a corresponding attribute of another training data distribution different from the multiple training data distributions. The method may further include generating an output matrix by the machine learning model and based on the input matrix and the domain index matrix, the output matrix comprising a corresponding output value generated based on (i) the corresponding input value and (ii) the corresponding domain index value, such that the corresponding output value exhibits the properties of the corresponding training data distribution.

[0004] In a second example embodiment, the system may include a processor and a non-transitory computer-readable medium thereon storing instructions which, when executed by the processor, cause the processor to perform operations according to the first example embodiment.

[0005] In the third example embodiment, instructions may be stored on a non-transitory computer-readable medium, which, when executed by a computing device, cause the computing device to perform the operations according to the first example embodiment.

[0006] In the fourth example embodiment, the system may include various means for performing each operation of the first example embodiment.

[0007] These and other embodiments, aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description and referring to the appropriate accompanying drawings. Furthermore, the content of this invention and the other descriptions and drawings provided herein are intended to illustrate embodiments by way of example only, and therefore many variations are possible. For example, structural elements and process steps may be rearranged, combined, distributed, eliminated, or otherwise modified while remaining within the scope of the claimed embodiments. Attached Figure Description

[0008] Figure 1 A computing device according to an example described herein is shown.

[0009] Figure 2 A computing system based on an example described herein is shown.

[0010] Figure 3 The training data distribution based on the example described in this article is shown.

[0011] Figure 4 A system based on an example described in this article is shown.

[0012] Figure 5 The input matrix, field index matrix, and output matrix are shown in the example described herein.

[0013] Figure 6 A training system based on an example described herein is shown.

[0014] Figure 7A An image is shown as an example based on the description in this article.

[0015] Figure 7B , Figure 7C , Figure 7D and Figure 7E A depth image is shown based on an example described in this article.

[0016] Figure 8 This is a flowchart based on the example described in this article. Detailed Implementation

[0017] This document describes exemplary methods, apparatus, and systems. It should be understood that the terms “exemplary” and “illustrative” are used herein to mean “serving as an example, instance, or illustration.” Unless stated otherwise, any embodiment or feature described herein as “exemplary,” “illustrative,” and / or “illustrative” is not necessarily to be construed as preferred or advantageous relative to other embodiment features. Therefore, other embodiments may be used and other changes may be made without departing from the scope of the subject matter presented herein.

[0018] Therefore, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that the aspects of this disclosure, as generally described herein and illustrated in the accompanying drawings, can be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0019] Furthermore, unless the context otherwise requires, the features shown in each figure can be used in combination with each other. Therefore, the figures should generally be considered as aspects of one or more overall embodiments, and it should be understood that not all features shown are necessary for every embodiment.

[0020] Additionally, any enumeration of elements, blocks, or steps in this specification or claims is for clarity purposes only. Therefore, such enumeration should not be construed as requiring or implying that these elements, blocks, or steps follow a particular arrangement or are performed in a particular order. Unless otherwise stated, the drawings are not to scale.

[0021] I. Overview

[0022] A machine learning model can be trained to perform a task using a corresponding training dataset (i.e., a training data distribution). This training dataset includes multiple pairs of training inputs and corresponding ground-truth outputs. In some cases, multiple different training datasets may be available for a given task, and the properties of these training datasets may differ from one another. For example, a first training dataset may include accurate ground-truth data for one or more first conditions associated with the task, while a second training dataset may include ground-truth data that is inaccurate for one or more first conditions but accurate for one or more second conditions. This difference between training datasets may be the result of various practical constraints associated with obtaining accurate ground-truth data, including the characteristics of the sensors used to generate the training data, the characteristics of the algorithms used to process sensor data and / or generate the training data, differences among human annotators of the training data, and / or the frequency of occurrence of different conditions represented by the training data, among other possible factors.

[0023] Training a machine learning model using the union of multiple training datasets without explicitly distinguishing them may cause the model to learn to generate an output that approximates the average of the true values ​​of these training datasets. However, when the true values ​​for a given condition differ across the training datasets, the average result may be inaccurate, causing the model to generate inaccurate results under certain conditions. Therefore, the machine learning model can instead be trained to allow the use of a domain index matrix to explicitly indicate at inference time one or more training datasets (on which the model has previously been trained) to use when generating output data. Specifically, during training, the model can be instructed to obtain the training dataset corresponding to its training samples. Thus, at inference time, the model can be configured to generate outputs with the same properties as the training dataset specified by the domain index matrix.

[0024] Specifically, a domain index matrix can include multiple elements, each of which can be associated with one or more domain index values. The one or more domain index values ​​of a particular element can include multiple values, for example, equal to the number of different training data distributions used in training the machine learning model. Each matrix element of the domain index matrix, and therefore its corresponding one or more domain index values, can correspond to one or more input values ​​of the input matrix. For example, the domain index value of a given matrix element of the domain index matrix can control how one, two, four, eight, or sixteen corresponding elements of the input matrix are processed to generate the corresponding output value of the output matrix. Thus, one or more domain index values ​​can instruct the machine learning model which one or more training data distributions to model and / or mimic when generating output values ​​based on one or more input values. Therefore, by specifying the degree to which different training data distributions affect a given pixel and / or a given group of pixels in the output matrix using a domain index matrix, the properties of the output matrix can be controlled pixel-wise and / or pixel-group-wise.

[0025] In some implementations, a domain detection model can be combined with training the machine learning model. The domain detection model can be used during training to determine the training domain index matrix based on the training input matrix and the ground truth index matrix. The machine learning model can then be used to generate a training output matrix based on the training domain index matrix and the corresponding training input matrix. Training the machine learning model using the training domain index matrix allows the model to learn to better handle noise, aggregate information from different training data distributions, and / or generate outputs that mix attributes from different training data distributions.

[0026] Furthermore, as a result of training, a machine learning model can be configured to generate multiple distinct output matrices based on a single input matrix, each associated with a corresponding domain index matrix. Therefore, by adjusting the values ​​of the domain index matrices, rather than retraining the machine learning model, it is possible to control a single machine learning model to deterministically generate outputs with different attributes. Additionally, compared to the sum of multiple machine learning models trained independently using corresponding training data distributions, the machine learning model can be smaller and therefore more computationally efficient (e.g., faster training, retrieval from memory, and / or execution). Moreover, because the machine learning model aggregates information from multiple different training data distributions, it can generate more accurate and / or higher-quality predictions than any single, independently trained model. Therefore, a machine learning model can be stored in a smaller memory space than would be required to store multiple machine learning models, yet still provide improved predictions relative to individually trained models. Thus, machine learning models can be deployed on memory-constrained devices where individually trained models cannot be deployed.

[0027] II. Example Computing Devices and Systems

[0028] Figure 1 An example computing device 100 is shown. The computing device 100 is shown in the form factor of a mobile phone. However, the computing device 100 can alternatively be implemented as a laptop computer, tablet computer, and / or wearable computing device, etc. The computing device 100 may include various components such as a body 102, a display 106, and buttons 108 and 110. The computing device 100 may further include one or more cameras, such as a front-facing camera 104 and a rear-facing camera 112.

[0029] The front-facing camera 104 may be positioned on the side of the body 102 that is typically facing the user during operation (e.g., on the same side as the display 106). The rear-facing camera 112 may be positioned on the side of the body 102 opposite to the front-facing camera 104. Referring to the cameras as front and rear is arbitrary, and the computing device 100 may include multiple cameras positioned on various sides of the body 102.

[0030] Display 106 may represent a cathode ray tube (CRT) display, a light-emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light-emitting diode (OLED) display, or any other type of display known in the art. In some examples, display 106 may display a digital representation of the current image captured by front camera 104 and / or rear camera 112, images that may be captured by one or more of these cameras, recently captured images by one or more of these cameras, and / or modified versions of one or more of these images. Thus, display 106 may act as a viewfinder for the camera. Display 106 may also support touchscreen functionality that enables adjustment of settings and / or configurations of one or more aspects of computing device 100.

[0031] The front-facing camera 104 may include an image sensor and associated optical elements, such as lenses. The front-facing camera 104 may provide zoom capability or may have a fixed focal length. In other examples, interchangeable lenses may be used with the front-facing camera 104. The front-facing camera 104 may have a variable mechanical aperture and a mechanical and / or electronic shutter. The front-facing camera 104 may also be configured to capture still images, video images, or both. Furthermore, the front-facing camera 104 may represent, for example, a single-field-of-view camera, a stereo camera, or a multi-field-of-view camera. The rear-facing camera 112 may be arranged similarly or differently. Additionally, one or more of the front-facing camera 104 and / or the rear-facing camera 112 may be an array of one or more cameras.

[0032] One or more of the front camera 104 and / or the rear camera 112 may include an illumination component that provides a light field to illuminate a target object or be associated with such an illumination component. For example, the illumination component may provide flash illumination or constant illumination of the target object. The illumination component may also be configured to provide a light field including one or more of structured light, polarized light, and light with specific spectral content. In the context of the examples herein, other types of light fields known for recovering three-dimensional (3D) models from objects are possible.

[0033] The computing device 100 may also include an ambient light sensor that can continuously or intermittently determine the ambient brightness of the scene that the cameras 104 and / or 112 can capture. In some implementations, the ambient light sensor may be used to adjust the display brightness of the display 106. Additionally, the ambient light sensor may be used to determine the exposure length of one or more of the cameras 104 or 112, or to assist in that determination.

[0034] The computing device 100 can be configured to capture images of a target object using a display 106 and a front-facing camera 104 and / or a rear-facing camera 112. The captured images can be multiple still images or a video stream. Image capture can be triggered by activating button 108, pressing a softkey on the display 106, or through some other mechanism. Depending on the implementation, images can be captured automatically at specific time intervals, such as when button 108 is pressed, under appropriate lighting conditions on the target object, when the computing device 100 is moved a predetermined distance, or according to a predetermined capture schedule.

[0035] Figure 2 This is a simplified block diagram illustrating some components of an example computing system 200. As an example and not a limitation, computing system 200 can be a cellular mobile phone (e.g., a smartphone), a computer (such as a desktop computer, laptop computer, tablet computer, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a game console, a robotic device, a vehicle, or some other type of device. Computing system 200 can represent, for example, aspects of computing device 100.

[0036] like Figure 2 As shown, the computing system 200 may include a communication interface 202, a user interface 204, a processor 206, a data storage 208, and a camera component 224, all of which can be communicatively linked together via a system bus, network, or other connection mechanism 210. The computing system 200 may be equipped with at least some image capture and / or image processing capabilities. It should be understood that the computing system 200 may represent a physical image processing system, a specific physical hardware platform on which image sensing and / or processing applications run in software, or other combinations of hardware and software configured to perform image capture and / or processing functions.

[0037] Communication interface 202 allows computing system 200 to communicate with other devices, access networks, and / or transmission networks using analog or digital modulation. Therefore, communication interface 202 can facilitate circuit-switched and / or packet-switched communications, such as Common Old Telephone Service (POTS) communications and / or Internet Protocol (IP) or other packet communications. For example, communication interface 202 may include a chipset and antenna arranged for wireless communication with a radio access network or access point. Furthermore, communication interface 202 may take the form of a wired interface or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. Communication interface 202 may also take the form of a wireless interface or include a wireless interface, such as Wi-Fi. Global Positioning System (GPS) or wide-area radio interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols can be used on communication interface 202. Furthermore, communication interface 202 may include multiple physical communication interfaces (e.g., Wi-Fi interfaces, etc.). Interface and wide area wireless interface).

[0038] User interface 204 can be used to allow computing system 200 to interact with human or non-human users, such as receiving input from the user and providing output to the user. Therefore, user interface 204 may include input components such as a keypad, keyboard, touch panel, computer mouse, trackball, joystick, microphone, etc. User interface 204 may also include one or more output components, such as a display screen, which may be combined with the touch panel, for example. The display screen may be based on CRT, LCD, and / or LED technology, or other technologies now known or developed in the future. User interface 204 may also be configured to generate one or more audible outputs via speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. User interface 204 may also be configured to receive and / or capture audible speech, noise, and / or signals via a microphone and / or other similar devices.

[0039] In some examples, the user interface 204 may include a display that serves as a viewfinder for still camera and / or video camera functions supported by the computing system 200. Additionally, the user interface 204 may include one or more buttons, switches, knobs, and / or dials that facilitate the configuration and focusing of camera functions and image capture. Some or all of these buttons, switches, knobs, and / or dials may be implemented via a touch-sensitive panel.

[0040] Processor 206 may include one or more general-purpose processors—such as microprocessors—and / or one or more special-purpose processors—such as digital signal processors (DSPs), graphics processing units (GPUs), floating-point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some cases, special-purpose processors may be capable of image processing, image alignment, and image merging, etc. Data storage 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated wholly or partially with processor 206. Data storage 208 may include removable and / or non-removable components.

[0041] Processor 206 may be able to execute program instructions 218 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 208 to perform the various functions described herein. Therefore, data storage 208 may include a non-transitory computer-readable medium having program instructions stored thereon that, when executed by computing system 200, cause computing system 200 to perform any methods, processes, or operations disclosed in this specification and / or the accompanying drawings. Execution of program instructions 218 by processor 206 may cause processor 206 to use data 212.

[0042] For example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more application programs 220 (e.g., camera functionality, address book, email, web browsing, social networking, audio-to-text functionality, text translation functionality, and / or game applications) installed on computing system 200. Similarly, data 212 may include operating system data 216 and application data 214. Operating system data 216 may be primarily accessed by operating system 222, and application data 214 may be primarily accessed by one or more of the application programs 220. Application data 214 may be located in a file system that is visible or hidden from the user of computing system 200.

[0043] Application 220 can communicate with operating system 222 through one or more application programming interfaces (APIs). These APIs can facilitate, for example, application 220 reading and / or writing application data 214, sending or receiving information via communication interface 202, receiving and / or displaying information on user interface 204, and so on.

[0044] In some cases, application 220 may be simply referred to as "app". Additionally, application 220 may be downloaded to computing system 200 through one or more online app stores or app markets. However, applications may also be installed on computing system 200 in other ways, such as via a web browser or through a physical interface of computing system 200 (e.g., a USB port).

[0045] Camera component 224 may include, but is not limited to, an aperture, shutter, recording surface (e.g., photographic film and / or image sensor), lens, shutter button, infrared projector, and / or visible light projector. Camera component 224 may include components configured to capture images in the visible spectrum (e.g., electromagnetic radiation with wavelengths of 380-700 nanometers) and / or components configured to capture images in the infrared spectrum (e.g., electromagnetic radiation with wavelengths of 701 nanometers to 1 millimeter), etc. Camera component 224 may be controlled at least in part by software executed by processor 206.

[0046] III. Example Training Data Distribution

[0047] Figure 3 Multiple training data distributions are shown, each of which can be used to train a machine learning model to perform a specific task. Specifically, Figure 3 The training data includes distributions 300, 320, and 340, which can be collectively referred to as training data distributions 300-340. Each training data distribution may include multiple corresponding training samples. Specifically, training data distribution 300 may include training sample 302, training data distribution 320 may include training sample 322, and training data distribution 340 may include training sample 342. Each corresponding training sample of training samples 302, 322, and 342 may include a corresponding training input and a corresponding ground truth output.

[0048] Training data distributions 300-340 may differ from each other on one or more attributes. Specifically, training data distribution 300 may be associated with attribute 304, training data distribution 320 may be associated with attribute 324, and training data distribution 340 may be associated with attribute 344. In some cases, attributes 304, 324, and 344 may be explicitly represented and / or indicated as part of the corresponding training data distribution. In other cases, attributes 304, 324, and 344 may inherently exist as part of training samples 302, 322, and 342, respectively, but may not be explicitly represented and / or indicated. Therefore, attributes 304, 324, and 344 are indicated by dashed lines.

[0049] Attribute 304 may include, for example, truth data accuracy 306, frequency content 308, statistical properties 310, feature class 312, and data source 314, etc. Similarly, attribute 324 may include truth data accuracy 326, frequency content 328, statistical properties 330, feature class 332, and data source 334, while attribute 344 may include truth data accuracy 346, frequency content 348, statistical properties 350, feature class 352, and data source 354.

[0050] The ground truth accuracy 306 can indicate the accuracy or correctness of the ground truth data represented by sample 302. In some cases, the ground truth accuracy 306 can vary based on and / or according to one or more other attributes. The frequency content 308 can indicate the frequencies represented by the corresponding training input and / or the ground truth output represented by training sample 302. In one example, when training sample 302 represents image data, the frequency content 308 can represent the spatial frequency content present in the image data. In another example, when training sample 302 represents audio data, the frequency content 308 can represent the audio content.

[0051] Statistical property 310 may indicate one or more statistical properties of the corresponding training input and / or true output represented by training sample 302. Statistical property 310 may include the mean, median, mode, range, variance, and / or standard deviation of various aspects of sample 302. Feature class 312 may indicate one or more classifications represented by training sample 302 based on features present in training sample 302. Features may include, for example, objects, sounds, words, and / or patterns thereof, and may depend on the type of data represented by training sample 302 (e.g., images, audio, text, etc.). Data source 314 may indicate the source from which training sample 302 is obtained and / or through which training sample 302 is generated. Data source 314 may indicate, for example, the type of sensor involved in generating sample 302, the algorithm involved in generating sample 302, and / or a unique identifier associated with training data distribution 300, etc.

[0052] In one example, training data distributions 300, 320, and 340 allow machine learning models to learn to generate depth images based on corresponding single-view images. Training samples 302, 322, and 342 may each include multiple pairs of (i) corresponding training single-view images (e.g., red-green-blue (RGB) images) and (ii) corresponding ground truth depth images. The corresponding ground truth depth images may have been generated for each training sample in one of several ways, such as based on one of several different input image data that allows for the determination of depth information and / or by processing the input image data using one of several different depth determination algorithms to determine the depth data. For example, for training sample 302, the ground truth depth image may have been generated based on stereo image data and / or disparity data obtained from one or more stereo cameras. For training sample 322, the ground truth depth image may have been generated based on multi-pixel (e.g., dual-pixel) images generated by one or more multi-pixel cameras. For training sample 342, ground truth depth images may have been generated based on the processing of multiple single-view images by another machine learning model, such as the MiDaS model discussed in the paper “Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer”, authored by Ranflt et al. and published as arXiv:1907.01341v3.

[0053] Therefore, training sample 302 may include a ground truth depth image containing high-frequency content, and thus represent the depth of various features in detail. Training sample 302 may also include noise, and therefore may not accurately represent the depth of some portions of various features. Training sample 322 may include a ground truth image that accurately represents the depth of texture features (e.g., image regions with high-frequency content), but may be inaccurate for features and / or regions with little or no texture (e.g., image regions with low-frequency content). Training sample 342 may include a ground truth image, which is generally accurate, but represents depth at a relatively low frequency, and therefore lacks high-frequency detail.

[0054] Therefore, each of the training data distributions 300, 320, and 340 may be relatively accurate in the first set of environments / conditions and relatively less accurate in the second set of environments / conditions. A machine learning model trained using training data distribution 300 can generate outputs that represent the properties of training data distribution 300, a machine learning model trained using training data distribution 320 can generate outputs that represent the properties of training data distribution 320, and a machine learning model trained using training data distribution 340 can generate outputs that represent the properties of training data distribution 340.

[0055] A machine learning model trained using a combined dataset representing the union of training data distributions 300, 320, and 340 can learn to predict an approximate average of the true values ​​associated with these data distributions. This can be undesirable because if the two training data distributions are significantly different—that is, one distribution represents the accurate true output and the other represents the inaccurate true output—the resulting average may still be inaccurate. Instead of simply training the machine learning model using the combined dataset, the model can be trained to know which distribution a given training sample belongs to, and thus deterministically control the degree to which different parts of the output are based on the different training data distributions.

[0056] IV. Example System for Machine Learning Models with Domain Selection

[0057] Figure 4 System 400 is illustrated, which can be configured to allow control over how a machine learning model uses multiple different training data distributions to generate output. Specifically, system 400 may include a machine learning model 404 and a domain index matrix generator 406. System 400 can be configured to generate an output matrix 410 based on an input matrix 402. System 400 can be implemented as hardware, software, or a combination thereof. For example, system 400 may be implemented by computing device 100 and / or computing system 200.

[0058] Input matrix 402 and / or output matrix 410 can represent various types of data, including, for example, images, point clouds, text, and / or waveforms, as well as other types of data that can be represented as matrices. For example, input matrix 402 can represent an input image (e.g., multiple values ​​associated with pixels of the input image), while output matrix 410 represents an output image generated based on the input image, which includes content beyond the content of the input image. Input matrix 402 and output matrix 410 can be considered representative examples of tensors with one or higher rank. For example, matrices 402 and / or 410 can have dimensions such as Hx1 (i.e., column vector), 1xW (i.e., row vector), HxW (i.e., matrix or rank 2 tensor), HxWxD (i.e., rank 3 tensor), etc. The dimensions of input matrix 402 and / or output matrix 410 can depend on factors such as the data type thus represented, the architecture / structure of machine learning model 404, and / or the task that machine learning model 404 is performing.

[0059] A machine learning model 404 may have been trained using multiple training data distributions (e.g., 300, 320, and 340) to generate an output matrix 410 based on an input matrix 402 and a domain index matrix 408. Specifically, the domain index matrix 408 allows selection of one or more training data distributions to be used in generating different portions of the output matrix 410. That is, using the domain index matrix 408, the machine learning model 404 can be controlled to generate, for example, a first portion of the output matrix 410 to represent the properties of a first training data distribution, and a second portion of the output matrix 410 to represent the properties of different training data distributions. Each of the multiple training data distributions can be considered to define its own domain and can therefore be identified by and / or associated with a domain index value.

[0060] Domain index matrix generator 406 can be configured to generate domain index matrix 408, which may include multiple domain index values. For each corresponding input value of input matrix 402, domain index matrix 408 may include a corresponding domain index value indicating the corresponding training data distribution to be used when generating the output value corresponding to the corresponding input value. The corresponding training data distribution may be one of multiple training data distributions used to train machine learning model 404. Each domain index value of domain index matrix 408 may be associated with one or more corresponding input values ​​of input matrix 402. Therefore, domain index matrix 408 can control which training data distribution to use when generating output matrix 410 based on per pixel (one domain index value controls one corresponding output value) or per group of pixels (one domain index value controls two or more corresponding output values).

[0061] In some implementations, the corresponding field index value can be, for example, a single integer value indicating a single corresponding training data distribution. For instance, the field index value could be 1, 2, or 3, indicating training data distributions 300, 320, or 340, respectively. Therefore, the corresponding field index value can be used to select the single training data distribution to be used when generating output values ​​corresponding to the corresponding input values. For example, when a given element of the field index matrix 408 is assigned the field index value 2, the machine learning model 404 can be configured to use training data distribution 320, instead of training data distributions 300 or 340, to generate the output value of the output matrix 410 based on the input values ​​of the input matrix 402 corresponding to the given elements of the field index matrix 408.

[0062] In other implementations, the corresponding domain index value may include multiple domain values, each of which indicates the extent to which the corresponding training data distribution is used in generating the output value corresponding to the corresponding input value. Therefore, each element of the domain index matrix 408 may be associated with multiple domain index values, rather than a single domain index value. For example, each element of the domain index matrix 408 may include a first domain index value, a second domain index value, and a third domain index value, indicating the extent to which training data distributions 300, 320, and 340 are used in generating the corresponding output value, respectively. Thus, the multiple domain index values ​​of each element can specify a mixture of training data distributions 300, 320, and 340 to be executed by the machine learning model 404 to generate the output value corresponding to the corresponding input value.

[0063] In some implementations, the domain index matrix generator 406 may include a user interface configured to allow a user to specify at least a portion of the domain index matrix 408. For example, a user may identify one or more attributes of the input matrix 402 based on a manual inspection of it (e.g., by viewing an image represented by the input matrix 402) and specify one or more domain index values ​​for the domain index matrix 408 based on these attributes. The user can select these domain index values ​​such that a distribution of training data associated with one or more attributes and / or similar attributes is used to generate the corresponding portion of the output matrix 410. By selecting different sets of domain index values, the user can use the machine learning model 404 to generate different output matrices based on a single input matrix without retraining the machine learning model 404.

[0064] In other implementations, the domain index matrix generator 406 can be configured to automatically determine one or more attributes of the input matrix 402 based on the input matrix 402, and generate one or more domain index values ​​for the domain index matrix 408 based on these attributes. For example, the domain index matrix generator 406 can be configured to determine and / or approximate the frequency content, statistical characteristics, feature classes, and / or data sources of the input matrix 402. The domain index matrix generator 406 can generate one or more domain index values ​​such that training data distributions associated with one or more attributes and / or similar attributes are used to generate corresponding portions of the output matrix 410, while training data distributions with different attributes are not used to generate corresponding portions of the output matrix 410.

[0065] Figure 5 Visual examples of input matrix 402, field index matrix 408, and output matrix 410 are provided. Specifically, Figure 5 This includes an example of a 6×6 input matrix 502 providing an input matrix 402, an example of a 3×3 field index matrix 508 providing a field index matrix 408, and an example of a 6×6 output matrix 510 providing an output matrix 410. The input matrix 502 includes matrix elements 520, 522, 524, and 526 with corresponding input values. The output matrix 510 includes matrix elements 540, 542, 544, and 546 with corresponding output values. The field index matrix 508 includes matrix element 530, which, in the example shown, includes multiple field index values ​​532.

[0066] In the example shown, each corresponding element of the domain index matrix 508 (and its associated domain index value) maps to four corresponding elements of the input matrix 502 and four corresponding elements of the output matrix 510, as shown by the shading of these elements. Specifically, the value of matrix element 530 of the domain index matrix 508 indicates to the machine learning model 404, based on the input values ​​of matrix elements 520, 522, 524, and 526, which training data distributions will be used when generating the output values ​​of matrix elements 540, 542, 544, and 546. In some implementations, the domain index matrix 508 may have a different size relative to the input matrix 502 and / or the output matrix 510. Therefore, the value of each matrix element of the domain index matrix 508 may correspond to fewer (e.g., 1) or more (e.g., 8) matrix elements of the input matrix 502 and / or the output matrix 510.

[0067] The domain index value 532 can include values ​​DI1, DI2, and DI3 to DI N (That is, the domain index value DI1-DI) NEach of these can be associated with a corresponding training data distribution. Specifically, the domain index value DI1 can correspond to a first training data distribution (e.g., training data distribution 300), the domain index value DI2 can correspond to a second training data distribution (e.g., training data distribution 320), the domain index value DI3 can correspond to a third training data distribution (e.g., training data distribution 340), and the domain index value DI... N This can correspond to the distribution of the Nth training data.

[0068] Domain index values ​​DI1-DI N Each of them can be from a predetermined range (e.g., R) MIN =0 to R MAX The first and second field index values ​​can be selected from (i.e., 540, 542, 544, and 546), thus indicating the extent to which the corresponding training data distribution influences the corresponding output values. For example, a field index value of 0 can indicate that the corresponding training data distribution will be used minimally (e.g., not at all), while a field index value of 1 can indicate that the corresponding training data distribution will be used to the maximum extent. Therefore, the ratio between the first and second field index values ​​can indicate the relative contribution of the corresponding training data distribution to the output value.

[0069] In some implementations, the machine learning model 404 can be configured to use the domain index values ​​DI1-DI N The sum is the predetermined value (i.e., Where V represents the operation under the condition of a predetermined value. The predetermined value can be based on a predetermined range. In one example, the predetermined value can be equal to the high end of the predetermined range R. MAX In other words, (For example, a predetermined value of 1 could correspond to the range 0 to 1). Therefore, it may be disallowed for multiple field index values ​​to simultaneously have a maximum value. In another example, a predetermined value could be chosen such that R... MAX <V≤NR MAX Therefore, the ratio DI of a given domain index value to a predetermined value... i / V can represent a portion of the total signal contributed by the corresponding output value from the corresponding training data distribution.

[0070] In other implementations, machine learning model 404 can be configured to be independent of the domain index values ​​DI1-DI. N The sum is operated on under the condition of a predetermined value. That is, machine learning model 404 can be configured to generate domain index values ​​DI1-DI. N The valid output value for any combination of domain index values. Therefore, the ratio of a given domain index value to the sum of its domain index values. It can represent a portion of the total signal contributing to the corresponding output value by the corresponding training data distribution.

[0071] In some implementations, the dimension of the domain index matrix 508 may correspond to the dimension of the latent space of the machine learning model 404. For example, the machine learning model 404 may include an encoder-decoder architecture, and the latent space may correspond to the output of the encoder, which is provided as input to the decoder. Therefore, the domain index matrix 508 may be provided as input to the decoder (along with the latent space representation of the input matrix 502), but may not be processed by the encoder. Figure 5 In the example, the latent space can have a 3×3 dimension, and the input matrix 502 can therefore be represented as a 3×3 matrix within the latent space. Thus, matrix element 530 can correspond to a single matrix element in the latent space representation of the input matrix 502, where this single matrix element collectively represents the values ​​of matrix elements 520, 522, 524, and 526. In other implementations, the domain index matrix 508 can have the same dimension as the input matrix 502, and therefore can be provided as input to the encoder and thus processed by the encoder.

[0072] V. Example Training System and Process

[0073] Figure 6 An example training system and process for machine learning model 404 are illustrated. Specifically, training system 600 may include machine learning model 404, domain detection model 606, prediction loss function 616, domain loss function 612, and model parameter tuner 630. Training system 600 may be configured to generate updated model parameters 632 for domain detection model 606 and / or machine learning model 404 based on training samples 620. Training system 600 may be implemented as hardware, software, or a combination thereof. For example, training system 600 may be implemented by computing device 100 and / or computing system 200.

[0074] Specifically, training sample 620 may include training input matrix 622, truth output matrix 624, and truth domain index matrix 626. In some implementations, the truth domain index matrix 626 may not be explicitly provided as part of training sample 620, but may be determined by training system 600 based on the corresponding training data distribution from which training sample 620 is selected. Training sample 620 may be a representative example of training samples 302, 322, and / or 342.

[0075] Domain detection model 606 may include a machine learning model configured to generate a training domain index matrix 608 based on a training input matrix 622 and a ground truth output matrix 624. During training, domain detection model 606 can learn to determine the training domain index matrix 608 such that it represents the training data distribution from which the selected training samples 620 are derived with increasing accuracy. Domain loss function 612 may be configured to generate a domain loss value 614 based on a comparison between the training domain index matrix 608 and the ground truth domain index matrix 626. Therefore, domain loss function 612 may be configured to quantify the accuracy with which domain detection model 606 identifies the training data distribution from which the selected training samples 620 are derived.

[0076] The domain loss function 612 can be expressed as, for example, Where x iterates over all elements of the truth domain index matrix 626, and f iterates over all domain index values ​​of a given element x. This represents the specific training domain index value of element x, which is training domain index 608. This represents the corresponding truth domain index value of the truth domain index matrix 626. In some implementations, the domain loss function 612 may additionally or alternatively apply other functions. and / or For example, logarithms, reciprocals, exponents, maximum values, minimum values, scaling and / or different norms (e.g., L-1 norm), etc.

[0077] The machine learning model 404 can be configured to determine the training output matrix 610 based on the training input matrix 622 and the training domain index matrix 608. The prediction loss function 616 can be configured to generate a prediction loss value 618 based on a comparison between the training output matrix 610 and the ground truth output matrix 624. Therefore, the prediction loss function 616 can be configured to quantify the accuracy of the machine learning model 404 in determining the ground truth output matrix 624 based on the training input matrix 622 and the output of the domain detection model 606.

[0078] The prediction loss function 612 can be expressed as follows: Where y iterates over all elements of the true output matrix 624, O 训练 (y) represents the training output value of element y in the training output matrix 610, and O GT (y) represents the corresponding true output value of the true output matrix 624. In some implementations, the prediction loss function 612 may additionally or alternatively apply other functions to O. 训练 (y) and / or O GT(y), for example, logarithm, reciprocal, exponential, maximum, minimum, scale, and / or different norms (e.g., L-1 norm), etc. For example, when the training output matrix 610 represents a depth image, the prediction loss function 616 can achieve scale-invariant depth / disparity loss and / or shift-invariant depth / disparity loss.

[0079] The model parameter adjuster 630 can be configured to determine updated model parameters 632 based on the domain loss value 614 and / or the prediction loss value 618. The model parameter adjuster 630 can also be configured to determine the total loss value based on a weighted sum of these loss values, which can be represented as L. 总 =L 域 +αL 预测 The value of α represents the relative weight of the corresponding loss value. The updated model parameters 632 may include one or more updated parameters of the machine learning model 404 and / or one or more updated parameters of the domain detection model 606.

[0080] The model parameter adjuster 630 can be configured to, for example, determine the total loss function L 总 The gradient is used to determine the updated model parameters 632. Based on this gradient and the total loss value, the model parameter adjuster 630 can be configured to select updated model parameters 632 that are expected to reduce the total loss value, and thus improve the performance of models 404 and / or 606. After applying the updated model parameters 632 to the machine learning model 404 and / or the domain detection model 606, another training iteration can be performed to compute another instance of the total loss value, and based on this, another instance of the updated model parameters 632 can be determined and applied to models 404 and / or 606 to further improve their performance. This training of models 404 and 606 can be repeated until, for example, the total loss value is reduced below a target threshold loss value.

[0081] Domain detection model 606 can be used to assist in training machine learning model 404, but may not be used during inference. Instead, as... Figure 4 As shown, during inference, the domain detection model 606 can be replaced by the domain index matrix generator 406. Specifically, by providing the training domain index matrix 608 instead of the ground truth domain index matrix 626 as input to the machine learning model 404 during training, the machine learning model 404 can learn to handle noise more robustly, use information from one training data distribution to improve predictions based on other training data distributions, and / or combine / mix properties from multiple different training data distributions during inference.

[0082] Specifically, the truth output matrix 624 can indicate the individual corresponding training data distribution to which the training sample 620 belongs (e.g., DI1 = 1 and...). Similarly, other ground truth output matrices may each indicate a single corresponding training data distribution. Conversely, the training domain index matrix 608 can indicate a mixture of different training data distributions (e.g., DI1-DI). N Two or more of these can be non-zero, because at least at the start of training, the output of the domain detection model 606 may be inaccurate. During training, the difference between the ground domain index matrix 626 and the training domain index matrix 608 may decrease, but they may not converge to exactly the same set of values. This exposes the machine learning model 404 to values ​​DI1-DI. N Various combinations of these can represent noise, while others can represent efficient combinations of the training data distribution.

[0083] Therefore, the machine learning model 404 can learn to predict a relatively accurate training output matrix 610 in the presence of noise and / or based on multiple different training data distributions. For example, where a portion of the training output 610 can be correctly generated using more than one underlying training data distribution, this training architecture allows the machine learning model 404 to learn to combine information from two training data distributions to generate the relevant portion of the training output matrix 610. Thus, when trained on three different training data distributions, the machine learning model 404 can perform more accurately and / or can have a smaller size than three separate models, each of which has been independently trained on its corresponding data distribution. Additionally, the machine learning model 404 can be configured to generate a continuous range of outputs representing different combinations of the three underlying datasets for a given input, rather than generating only three discrete outputs as three independently trained models would.

[0084] The machine learning model 404 can be stored in a memory portion. This memory portion is relatively small compared to the memory required to store an independently trained model. This memory portion can, for example, be part of the memory of a device with limited memory. The limited memory of the device may make storing and executing independently trained models difficult or impossible. That is, the machine learning model 404 may be able to be deployed on devices that cannot efficiently deploy independently trained models. The device can, for example, be a mobile computing device.

[0085] VI. Example Applications of Machine Learning Models

[0086] Figure 7A , Figure 7B , Figure 7C , Figure 7D and Figure 7E This indicates that machine learning model 404 will be applied to various aspects of the task of generating depth images based on single-view / monocular images. Specifically, Figure 7AImage 700 includes a scene containing grass and leaves. Region 702 contains relatively high-frequency textures, while region 704 contains relatively low-frequency textures. Due to the difference in frequency content in regions 702 and 704, different training data distributions may lead to different levels of accuracy in depth prediction for these regions.

[0087] Specifically, Figure 7B This includes a depth image 710 generated by a machine learning model based on image 700. This machine learning model has been specifically trained using a training data distribution (e.g., 300) that has ground truth depth images generated based on stereo image data and / or disparity data obtained from multiple pairs of stereo cameras. Depth image 710 has relatively high spatial frequency content. Additionally, the depth values ​​represented by depth image 710 are generally accurate for most of image 700; however, depth image 710 includes a relatively small number of pixels (e.g., less than 5% of image 710) that have incorrect depth values ​​that differ significantly from the actual depth values. That is, depth image 710 may be detailed but noisy.

[0088] Figure 7C This includes a depth image 720 generated by a machine learning model based on image 700. This machine learning model has been specifically trained using a training data distribution (e.g., 320) that has ground truth depth images generated based on multi-pixel (e.g., dual-pixel) images produced by one or more multi-pixel cameras. Depth image 720 may include accurate depth values ​​in regions with discernible texture, but may include erroneous depth values ​​in regions lacking discernible texture. Therefore, the depth value in region 702 of depth image 720 (e.g., because this region is in focus) may be more accurate than the depth value in region 704 of image 720 (e.g., because this region is out of focus). Depth image 720 may include less detail and less noise than depth image 710, and may be inaccurate in low-frequency (e.g., textureless) regions.

[0089] Figure 7DThis includes a depth image 730 generated from image 700 by a machine learning model specifically trained using a training data distribution (e.g., 340) that contains ground-value depth images generated by processing multiple single-viewpoint images using a MiDaS model / algorithm. Depth image 730 may include generally accurate depth values ​​regardless of the underlying spatial frequency content in image 700, but these depth values ​​may not represent high frequencies. Therefore, the depth values ​​in region 704 of depth image 730 may be more accurate than the corresponding depth values ​​in region 704 of image 720, and the depth values ​​in region 702 of depth image 730 may represent less detail than the corresponding depth values ​​in region 702 of image 720. Depth image 730 may contain less detail and noise than depth image 710.

[0090] Figure 7E This includes a depth image 740 generated by machine learning model 404 based on image 700, which has been used by training system 600. Figure 7B , Figure 7C and Figure 7D One of three associated training data distributions is used for training. Specifically, depth image 740 can be generated by spatially varying information from the three training data distributions based on the spatial frequency content of different parts of image 700. For example, domain index generator 406 can implement the following function to generate domain index matrix 408 used in determining depth image 740: Where g = gradient(image 700), g represents the spatial frequency content of image 700, λ is a modifiable scaling factor (λ = 0.45 in the example shown), and the clip() function is configured to scale g / λ to the interval [0.0, 1.0]. Furthermore, the domain index generator 406 can be configured to assign domain index values ​​based on DI1 = β / 2, DI2 = β / 2, and DI3 = 1.0 - β.

[0091] Therefore, the portion of depth image 740 corresponding to the high-frequency portion of image 700 can be primarily and equally based on... Figure 7B and Figure 7C The training data distribution is used to generate the image, and the portion of depth image 740 corresponding to the low-frequency portion of image 700 can be mainly based on... Figure 7D The training data distribution is used to generate the image, where the mid-frequency contains a mixture of all three training datasets. The value of λ and / or the function used when calculating DI1, DI2, and / or DI3 based on β can be changed to generate a depth image that includes the desired mixture of attributes from different training datasets. In other implementations, the domain index generator 406 can be additionally or alternatively configured to quantize other attributes of the input matrix 402 and, based on this, select the corresponding mixture of the underlying training data distribution.

[0092] VII. Additional Example Operations

[0093] Figure 8 A flowchart of operations is shown, which relate to generating output data through a machine learning model while deterministically selecting the distribution of training data on which the output data is based. These operations can be performed by computing device 100, computing system 200, system 400, and / or training system 600, etc. Figure 8 The embodiments can be simplified by removing any one or more features shown therein. Furthermore, these embodiments can be combined with any features, aspects, and / or implementations described in the previous figures or herein.

[0094] Box 800 may involve obtaining an input matrix representing the input data.

[0095] Box 802 may involve determining a domain index matrix that includes a corresponding domain index value for each corresponding input value of the input matrix, the corresponding domain index value indicating the corresponding training data distribution to be used in generating output values ​​corresponding to the corresponding input values. The corresponding training data distribution may be one of several training data distributions.

[0096] Box 804 may involve providing an input matrix and a domain index matrix to a machine learning model that has been trained to generate an output matrix using multiple training data distributions. Each of the multiple training data distributions may be associated with a corresponding attribute of another training data distribution that is different from the multiple training data distributions.

[0097] Box 806 may involve a machine learning model generating an output matrix based on an input matrix and a domain index matrix, the output matrix including a corresponding output value generated for each corresponding input value of the input matrix based on (i) the corresponding input value and (ii) the corresponding domain index value, such that the corresponding output value reflects the properties of the corresponding training data distribution.

[0098] In some embodiments, the properties of each corresponding training data distribution may include the accuracy of the ground truth data associated with one or more conditions represented by the corresponding training data distribution. The accuracy of the ground truth data may vary across multiple training data distributions as a function of one or more conditions.

[0099] In some embodiments, one or more conditions represented by the corresponding training data distribution may include one or more of the following: (i) the frequency content of the ground truth data, (ii) the frequency content of the training input corresponding to the ground truth data, (iii) the statistical properties of the ground truth data, or (iv) the statistical properties of the training input.

[0100] In some embodiments, the properties of each corresponding training data distribution may include one or more classifications of the features of the input data represented by the corresponding training data distribution. The one or more classifications may be a subset of multiple possible classifications of the features of the input data. Multiple possible classifications may be represented by multiple training data distributions.

[0101] In some embodiments, the corresponding domain index value may include multiple domain index values. Each corresponding domain index value may be selected from a predefined range and may indicate the range of the corresponding training data distribution to be used when generating an output value corresponding to the corresponding input value. The corresponding training data distribution for each corresponding domain index value may be one of multiple training data distributions.

[0102] In some embodiments, the number of domain index values ​​among multiple domain index values ​​may be equal to the number of training distributions among multiple training distributions.

[0103] In some embodiments, the sum of a plurality of domain index values ​​corresponding to a respective input value may be equal to a predetermined value. Each respective domain index value may indicate a portion of the total signal of the corresponding output value contributed by the corresponding training data distribution.

[0104] In some embodiments, a machine learning model may have been trained using a training process that includes obtaining training samples comprising a training input matrix and a ground truth output matrix. The training process may further include determining a training domain index matrix for the training samples, the training domain index matrix comprising a corresponding training domain index value for each corresponding training input value of the training input matrix, the corresponding training domain index value indicating (i) the training data distribution to be used in generating training output values ​​corresponding to the corresponding training input values ​​and (ii) the training data distribution to which the training samples belong. The training process may additionally include providing the machine learning model with the training input matrix and the training domain index matrix, and having the machine learning model generate a training output matrix based on the training input matrix and the training domain index matrix, the training output matrix comprising a corresponding training output value generated for each corresponding training input value of the training input matrix based on (i) the corresponding training input value and (ii) the corresponding training domain index value. The training process may further include determining a predictive loss value using a predictive loss function configured to compare the training input matrix and the training output matrix, and adjusting one or more parameters of the machine learning model based on the predictive loss value.

[0105] In some embodiments, determining the training domain index matrix may include determining the corresponding training data distribution to which the training samples belong, and assigning a training domain index value associated with the corresponding training data distribution to which the training samples belong to each corresponding training input value of the training input matrix.

[0106] In some embodiments, the training samples may further include a ground truth index matrix, which indicates the corresponding training data distribution to which the training samples belong. The training process may also include determining the training domain index matrix using a domain detection model and based on the corresponding training input matrix and the corresponding ground truth output matrix of the respective training samples. The training process may further include determining a domain loss value using a domain loss function configured to compare the training domain index matrix and the ground truth index matrix, and adjusting one or more parameters of at least one of (i) the machine learning model or (ii) the domain detection model based on the predicted loss value.

[0107] In some embodiments, the input matrix may include an input image. The output matrix may include an output image representing one or more of the following: (i) a bounding box associated with an object represented by the input image, (ii) a segmentation map of the object represented by the input image, (iii) one or more coordinates associated with the object represented by the input image, or (iv) a representation of the pose of the object represented by the input image.

[0108] In some embodiments, the input matrix may include an image, and the output matrix may include a depth map. Multiple training data distributions may include a first training data distribution generated based on multiple stereo images, a second training data distribution generated based on multiple multi-pixel images, and a third training data distribution generated based on the processing of multiple single-view images by another machine learning model.

[0109] In some embodiments, determining the domain index matrix may include determining the corresponding attribute value of a specific attribute of a plurality of training data distributions based on the input matrix and for each corresponding input value thereon, and determining the corresponding domain index value for each corresponding input value of the input matrix based on the corresponding attribute value.

[0110] In some embodiments, determining the domain index matrix may include providing a user interface configured to allow the definition of the domain index matrix, and receiving the definition of at least a portion of the domain index matrix through the user interface.

[0111] In some embodiments, a second field index matrix, different from the field index matrix, may be determined. The input matrix and the second field index matrix can be provided to a machine learning model. The machine learning model can then generate a second output matrix, different from the output matrix, based on the input matrix and the second field index matrix.

[0112] In some embodiments, the input matrix and the output matrix may each have a first dimension, and the domain index matrix may have a second dimension smaller than the first dimension and corresponding to the latent space of the machine learning model. Generating the output matrix may include generating a latent space representation of the input matrix. A given domain index value may correspond to (i) multiple input values ​​of the input matrix as represented in the first dimension, and (ii) a single latent space value that represents multiple input values ​​as part of the latent space representation of the input matrix in the second dimension.

[0113] VIII. Conclusion

[0114] This disclosure is not limited in its specific embodiments described herein, which are intended to be illustrative of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. In addition to those described herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0115] The detailed description above, with reference to the accompanying drawings, illustrates various features and operations of the disclosed systems, apparatus, and methods. In the drawings, similar symbols generally identify similar components unless the context otherwise indicates otherwise. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that aspects of this disclosure, as generally described herein and illustrated in the drawings, can be arranged, replaced, combined, separated, and designed in a variety of different configurations.

[0116] With respect to any or all message flow diagrams, scenarios, and flowcharts shown in the accompanying drawings and discussed herein, each step, block, and / or communication may represent information processing and / or information transmission according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed out of order with respect to the sequence shown or discussed (including substantially simultaneous or reverse order, depending on the functions involved). Furthermore, more or fewer blocks and / or operations may be associated with any message flow diagrams, scenarios, and flowcharts discussed herein. Figure 1 These message flow diagrams, scenarios, and flowcharts can be used together, and they can be partially or fully combined with each other.

[0117] A step or block representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, a block representing information processing may correspond to a module, segment, or portion of program code (including associated data). Program code may include one or more instructions executable by a processor for implementing a specific logical operation or action in the method or technique. Program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including random access memory (RAM), a disk drive, a solid-state drive, or other storage media.

[0118] Computer-readable media may also include non-transitory computer-readable media, such as short-term data storage media like register memory, processor cache, and RAM. Computer-readable media may also include long-term storage media for program code and / or data. Therefore, computer-readable media can include secondary or permanent long-term memory, such as read-only memory (ROM), optical discs or magnetic disks, solid-state drives, and optical disc read-only memory (CD-ROM). Computer-readable media can also be any other volatile or non-volatile storage system. Computer-readable media can be considered, for example, computer-readable storage media or tangible storage devices.

[0119] Furthermore, a step or block representing one or more information transfers may correspond to information transfers between software and / or hardware modules within the same physical device. However, other information transfers may occur between software and / or hardware modules in different physical devices.

[0120] Furthermore, the specific arrangements shown in the accompanying drawings should not be considered limiting. It should be understood that other embodiments may include, more or less, each element shown in the given drawings. Additionally, some of the shown elements may be combined or omitted. Also, exemplary embodiments may include elements not shown in the drawings.

[0121] While aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting, wherein the true scope is indicated by the following claims.

Claims

1. A computer-implemented method, comprising: Obtain an input matrix representing the input data, wherein the input matrix represents the input image data; Determine a domain index matrix, the domain index matrix including a corresponding domain index value for each corresponding input value of the input matrix, the corresponding domain index value indicating a corresponding training data distribution to be used when generating an output value corresponding to the corresponding input value, wherein the corresponding training data distribution is one of a plurality of training data distributions; The input matrix and the domain index matrix are provided to a machine learning model, which has been trained to generate an output matrix using the plurality of training data distributions, wherein each of the plurality of training data distributions is associated with an attribute of a corresponding attribute of another training data distribution that is different from the plurality of training data distributions; as well as An output matrix is ​​generated by the machine learning model based on the input matrix and the domain index matrix. The output matrix includes a corresponding output value generated for each corresponding input value of the input matrix based on (i) the corresponding input value and (ii) the corresponding domain index value, such that the corresponding output value reflects the properties of the corresponding training data distribution. The output matrix includes data describing the input image data or data describing the output image generated based on the input image data.

2. The computer-implemented method of claim 1, wherein the attribute of each corresponding training data distribution includes the accuracy of true data associated with one or more conditions represented by the corresponding training data distribution, and wherein the accuracy of the true data varies across the plurality of training data distributions as a function of the one or more conditions.

3. The computer-implemented method according to claim 2, wherein the one or more conditions represented by the corresponding training data distribution include one or more of the following: (i) the frequency content of the true data, (ii) the frequency content of the training input corresponding to the true data, (iii) the statistical characteristics of the true data, or (iv) the statistical characteristics of the training input.

4. The computer-implemented method according to any one of claims 1 to 3, wherein the attribute of each corresponding training data distribution includes one or more classifications of features of the input data represented by the corresponding training data distribution, wherein the one or more classifications are subsets of a plurality of possible classifications of features of the input data, and wherein the plurality of possible classifications are represented by the plurality of training data distributions.

5. The computer-implemented method of claim 1, wherein the corresponding domain index value comprises a plurality of domain index values, wherein each corresponding domain index value of the plurality of domain index values ​​is selected from a predefined range and indicates the extent to which a corresponding training data distribution is used in generating the output value corresponding to the corresponding input value, and wherein the corresponding training data distribution of each corresponding domain index value is one of the plurality of training data distributions.

6. The computer-implemented method according to claim 5, wherein the number of domain index values ​​among the plurality of domain index values ​​is equal to the number of training distributions among the plurality of training distributions.

7. The computer-implemented method of claim 5, wherein the sum of the plurality of domain index values ​​corresponding to the respective input values ​​is equal to a predetermined value, and wherein each respective domain index value indicates a portion of the total signal of the corresponding output value contributed by the corresponding training data distribution.

8. The computer-implemented method of claim 1, wherein the machine learning model has been trained using a training process, the training process comprising: Obtain training samples including the training input matrix and the ground truth output matrix; A training domain index matrix is ​​determined for the training samples, the training domain index matrix including a corresponding training domain index value for each corresponding training input value of the training input matrix, the corresponding training domain index value indicating (i) the training output value to be used in generating the training output value corresponding to the corresponding training input value and (ii) the corresponding training data distribution to which the training samples belong; The machine learning model is provided with the training input matrix and the training domain index matrix; The machine learning model generates a training output matrix based on the training input matrix and the training domain index matrix. For each corresponding training input value of the training input matrix, the training output matrix includes a corresponding training output value generated based on (i) the corresponding training input value and (ii) the corresponding training domain index value. The prediction loss value is determined using a prediction loss function configured to compare the training input matrix and the training output matrix; as well as Adjust one or more parameters of the machine learning model based on the predicted loss value.

9. The computer-implemented method according to claim 8, wherein determining the training domain index matrix comprises: Determine the corresponding training data distribution to which the training sample belongs; as well as Assign a training domain index value associated with the corresponding training data distribution to which the training sample belongs to each corresponding training input value of the training input matrix.

10. The computer-implemented method of claim 8, wherein the training sample further comprises a truth domain index matrix indicating the corresponding training data distribution to which the training sample belongs, and wherein the training process further comprises: The training domain index matrix is ​​determined by a domain detection model based on the corresponding training input matrix and the corresponding ground truth output matrix of the corresponding training samples. The domain loss value is determined using a domain loss function configured to compare the training domain index matrix and the truth domain index matrix; as well as Based on the predicted loss value, adjust one or more parameters of at least one of (i) the machine learning model or (ii) the domain detection machine learning model.

11. The computer-implemented method of claim 1, wherein the input matrix comprises an input image, and wherein the output matrix comprises an output image representing one or more of the following: (i) a bounding box associated with an object represented by the input image, (ii) a segmentation map of the object represented by the input image, (iii) one or more coordinates associated with the object represented by the input image, or (iv) a representation of the pose of the object represented by the input image.

12. The computer-implemented method of claim 1, wherein the input matrix comprises an image, and wherein the output matrix comprises a depth map, and wherein the plurality of training data distributions comprise: The first training data distribution generated from multiple stereo images; A second training data distribution generated based on multiple multi-pixel images; as well as The third training data distribution is generated based on the processing of multiple single-view images by another machine learning model.

13. The computer-implemented method according to claim 1, wherein determining the domain index matrix comprises: Based on the input matrix and its corresponding input value for each, determine the corresponding attribute value of a specific attribute of the plurality of training data distributions; as well as For each corresponding input value of the input matrix, the corresponding field index value is determined based on the corresponding attribute value.

14. The computer-implemented method of claim 1, wherein determining the domain index matrix comprises: Provide a user interface configured to allow the definition of the domain index matrix; as well as The user interface receives at least a portion of the definition of the domain index matrix.

15. The computer-implemented method according to claim 1, further comprising: Determine a second field index matrix that is different from the stated field index matrix; The machine learning model is provided with the input matrix and the second domain index matrix; as well as A second output matrix, different from the output matrix, is generated by the machine learning model based on the input matrix and the second domain index matrix.

16. The computer-implemented method of claim 1, wherein the input matrix and the output matrix each have a first dimension, wherein the domain index matrix has a second dimension smaller than the first dimension and corresponding to the latent space of the machine learning model, wherein generating the output matrix includes generating a latent space representation of the input matrix, and wherein a given domain index value corresponds to (i) a plurality of input values ​​of the input matrix as represented at the first dimension, and (ii) a single latent space value, the single latent space value representing at the second dimension the plurality of input values ​​as part of the latent space representation of the input matrix.

17. A system comprising: processor; as well as A non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, cause the processor to perform the operation according to any one of claims 1 to 16.

18. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a computing device, cause the computing device to perform operations according to any one of claims 1-16.

Images