User annotation of stereoscopic images

Abstract

Certain aspects of the present disclosure provide techniques for generating three-dimensional coherent training data for an image detection machine learning model. Embodiments include receiving a first image of an object from a first perspective and a second image of the object from a second perspective. Embodiments include receiving a user input identifying a location in the first image corresponding to a point on the object. Embodiments include displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image. Embodiments include generating training data for the machine learning model based on updated user input associated with the series of possible locations.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 821,627 filed on August 23, 2022, which is hereby incorporated by reference into this application. Background Art

[0003] Aspects of the present disclosure relate to machine learning.

[0004] Machine learning has revolutionized many aspects of computer vision. However, using machine learning to identify points on objects in image data in a three-dimensionally coherent manner remains a challenging computer vision task with many useful purposes. For example, while conventional techniques may allow the location of a fingertip to be identified in an image, the identified location may not be accurate from a three-dimensional perspective. For example, if another image of the fingertip is taken from a different perspective, the location of the fingertip in the other image may not be consistent with the location identified in the first image.

[0005] Training machine learning models for computer vision tasks is typically performed using supervised machine learning techniques, which require large amounts of prepared training data (e.g., training data with accurate pixel-level labels for image data). Unfortunately, in many real-world applications, such data is often unavailable and difficult to obtain, especially image data with three-dimensional coherent labels. Therefore, in practice, it is difficult or even impossible to train high-performance models for three-dimensional coherent identification of points on an object in many contexts.

[0006] Therefore, there is a need for improved machine learning techniques for identifying the locations of points on objects in images. Summary of the invention

[0007] Certain aspects provide a method comprising: receiving a first image of an object from a first perspective; receiving a second image of the object from a second perspective; receiving user input identifying a location in the first image corresponding to a point on the object; displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receiving updated user input, the updated user input identifying an updated location in the first image corresponding to the point on the object based on the series of possible locations excluding the point on the object in the second image; displaying an updated series of possible locations in the second image corresponding to the point on the object based on the updated location in the first image; and generating training data for a machine learning model based on the updated user input.

[0008] Other aspects provide: a processing system configured to perform the aforementioned methods and those described herein; a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods and those described herein; a computer program product embodied on a computer-readable storage medium comprising code for performing the aforementioned methods and those further described herein; and a processing system comprising components for performing the aforementioned methods and those further described herein.

[0009] The following description and associated drawings set forth in detail certain illustrative features of the one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings depict certain aspects of the one or more aspects and therefore are not to be considered limiting of the scope of the present disclosure.

[0011] Figures 1A to 1E is an illustration of example steps associated with generating 3D coherent training data for an image detection machine learning model.

[0012] Figure 2 Depicted are examples related to training of a machine learning model for three-dimensional coherent image detection according to embodiments of the present disclosure.

[0013] Figure 3 Depicts example operations associated with generating 3D coherent training data for an image detection machine learning model.

[0014] Figure 4 Depicted are devices adapted to perform operations for the techniques disclosed herein such as Figure 3 Examples of operating processing systems are depicted and described.

[0015] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation. DETAILED DESCRIPTION

[0016] Aspects of the present disclosure provide apparatus, methods, processing systems, and non-transitory computer-readable media for generating three-dimensional coherent training data for an image detection machine learning model.

[0017] Identifying the three-dimensional coherent position of a specific point on an object in an image is a valuable capability in computer vision applications, which can be used in augmented reality, simultaneous localization and mapping (SLAM), navigation, object detection, keypoint identification and tracking, and semantic segmentation, to name just a few examples. For example, determining the three-dimensional coherent position of a point on an object in an image is useful for depth perception, obstacle avoidance (e.g., for (semi-)autonomous flying drones, (semi-)autonomous or assisted driving cars, (semi-)autonomous operating warehouse robots, generally (semi-)autonomous moving household and other robots), 3D construction of the environment, spatial scene understanding, receiving user input via gestures, augmenting specific points on an object in an image with additional data, and / or other examples.

[0018] Machine learning models for computer vision tasks are typically trained through supervised training techniques based on labeled training data. For example, one or more users may annotate an image with the location of specific points of interest in the image, and the annotations may be used as labels for the training data. Images from the training data may be provided as input to the machine learning model, and one or more parameters of the machine learning model may be iteratively adjusted until one or more outputs from the model based on the input are consistent with the labels in the training data or until one or more conditions are satisfied.

[0019] A challenge in training such machine learning models is obtaining training data that is three-dimensionally coherent. For example, a user may annotate an image with the location of a particular point on an object, but small inaccuracies in the annotation (e.g., being off by just a few pixels from the exact location of the point) may result in the location not being three-dimensionally coherent. Reasons for such inaccuracies in the annotation include a lack of depth information in the image and complete or partial occlusion of a point in the image. For example, if a particular finger is hidden behind one or more other fingers in the image, it may be difficult to accurately annotate the image with the location of the fingertip of the particular finger, and the resulting lack of precision may result in the annotation not being three-dimensionally coherent.

[0020] Advantageously, embodiments of the present disclosure relate to utilizing stereo images to improve the accuracy of annotations during guidance, thereby generating three-dimensional coherent training data. Specifically, two images of an object are captured from different perspectives. Then, a first image of the two images is provided to a user to identify the position of a point on the object in the image. Once the user identifies the position in the first image, the position is used to determine a set of possible positions of the point on the object in the second image. For example, ray casting can be used to generate lines representing epipolar geometric constraints in the second image based on the position identified by the user in the first image. A set of possible positions of the point in the second image can be displayed to the user in the second image (e.g., represented by lines), and the user can determine whether the actual position of the point in the second image is included in a set of possible positions (e.g., whether the displayed line intersects with the actual position of the point in the second image). Then, the user can provide an input indicating whether the actual position of the point in the second image is included in a set of possible positions, and in some embodiments, if the actual position is included in the set, the actual position is identified.

[0021] If the user input indicates that the actual location of the point in the second image is included in the set of possible locations, the location identified by the user in the first image can be determined to be three-dimensionally coherent, and the annotated first image can be used as training data for the machine learning model. If the user provides the actual location of the point in the second image (e.g., by selecting a point on the displayed line), the annotated second image can also be used as training data for the machine learning model.

[0022] If the user input indicates that the actual location of the point in the second image is not included in the set of possible locations, the user may be prompted to provide an updated location of the point in the first image (e.g., because the originally identified location was not three-dimensionally coherent and therefore inaccurate). An updated set of possible locations for the point in the second image is then determined and displayed to the user for updated input. This process may be iteratively repeated until input is received from the user indicating that the actual location of the point in the second image is included in the displayed set of possible locations for the point in the second image. Once such input is received, the annotated first image (and in some embodiments the annotated second image) may be used as training data for the machine learning model.

[0023] It is worth noting that, while stereo images are used in embodiments of the present disclosure to improve the accuracy and three-dimensional coherence of image annotation, the resulting annotated images can be used as training data for a machine learning model for identifying the location of objects in a single (e.g., a single) image. Thus, by relying on stereo images to ensure the three-dimensional coherence of the training data, the techniques described herein allow a machine learning model to be trained to identify the three-dimensional coherent locations of points on objects in non-stereo images.

[0024] By ensuring the three-dimensional coherence of the annotations via a guided feedback loop, embodiments of the present disclosure produce more accurate training data, thereby leading to improved machine learning models that output more accurate results. Therefore, the techniques described herein constitute a significant technical improvement over existing machine learning techniques.

[0025] Example process for generating 3D coherent training data for machine learning models

[0026] Figure 1A Illustration 100A includes image acquisition stages associated with generating three-dimensional coherent training data for an image detection machine learning model.

[0027] Cameras 122 and 124 are used to capture two images 102 and 104 from two different perspectives of an object 110 (eg, a person's hand). Images 102 and 104 together constitute a stereoscopic image and represent different perspectives of the same object 110 captured simultaneously.

[0028] Figure 1B An illustration 100B of the image annotation phase associated with generating three-dimensional coherent training data for an image detection machine learning model is included.

[0029] In the example, image 102 is displayed to a user, and the user provides input identifying a location 130 of a point on object 110 (such as a fingertip on a hand) in image 102. For example, the user may select location 130 through touch input on a touch screen, by clicking location 130 using a cursor, or via some other form of input.

[0030] Figure 1C Illustration 100C includes a 3D coherence inspection phase associated with generating 3D coherence training data for an image detection machine learning model.

[0031] Lines 140 representing a set of possible locations of points on object 110 in image 104 are determined based on locations 130. For example, ray casting may be used to determine epipolar geometry constraints relative to image 102 based on locations 130 in image 104 and based on the relative positions of cameras 122 and 124.

[0032] When the relative positions of two cameras through which two images of an object are captured are known, certain conclusions can be drawn. For example, point x may represent a point on the object (e.g., a fingertip on a hand), x1 may represent the position of point x in a first image (e.g., image 102), and x2 may represent the position of point x in a second image (e.g., image 104). If point x1 is known (e.g., position 130), and the epipolar line corresponding to point x1 in the second image is known, then the projection of point x into the second image must lie on point x2 on that particular epipolar line. Thus, for every point observed in the first image, the same point must be observed in the other image on a known epipolar line. This provides an epipolar geometry constraint.

[0033] The epipolar line in the second image may be determined based on the point x1 and the relative positions of the cameras 122 and 124 (e.g., the optical center O1 of the lens of the first camera 122 and the optical center O2 of the lens of the second camera 124). Since the optical centers of the camera lenses are different, each center projects onto a different point in the image plane of the other camera. The two image points represented by e1 (representing the projection of the optical center O2 of the lens of the second camera 124 in the image plane of the first camera 122) and e2 (representing the projection of the optical center O1 of the lens of the first camera 122 in the image plane of the second camera 124) are referred to as epipolar points or epipolar points.

[0034] The line O1-x is considered a point by the first camera 122 because it is directly in line with the optical center of the lens of that camera. The second camera 124 considers this line as a line in its image plane. The line (e2-x2) in the second camera 124 is called the epipolar line. In some embodiments, the epipolar line e2-x2 can be determined using a rotation matrix and / or a translation vector that describes the relative pose between the two cameras 122 and 124, as is known in the art. In some embodiments, line 140 represents the epipolar line e2-x2 in the second image 104.

[0035] Image 104 may be displayed to the user along with line 140. In some embodiments, the user provides input indicating whether the location of a point on the object in image 104 is included in a set of possible locations represented by line 140. As shown in diagram 100C, if location 130 in image 104 is not accurate, a gap 142 may exist between line 140 and the actual location of the point on the object in image 102. This error indicates a lack of accuracy in location 130. Therefore, the user may provide input indicating that the actual location of the point on the object in image 104 is not included in the set of possible locations represented by line 140.

[0036] Figure 1DIllustration 100D of an updated image annotation phase including generation of three-dimensional coherent training data for an image detection machine learning model.

[0037] In an example, in response to receiving an indication Figure 1C The actual position of the point on the object in the image 104 is not included in the Figure 1C The line 140 represents a set of possible positions of the input, Figure 1C The image 102 is displayed to the user for updated annotation. The user provides input identifying an updated location 150 (replacing location 130) in the image 102 of a point on the object 110. For example, the user may select location 150 through touch input on a touch screen, by clicking on location 150 with a cursor, or via some other form of input.

[0038] Figure 1E Illustration 100E of an updated 3D coherence inspection phase associated with generating 3D coherence training data for an image detection machine learning model.

[0039] Based on the positions 150, a line 160 representing a set of possible positions of a point on the object 110 in the image 104 is determined. Figure 1C As described, line 160 may represent an epipolar geometry constraint with respect to image 104 based on location 150 in image 102 and based on the relative positions of cameras 122 and 124 .

[0040] Image 104 may be displayed to the user along with line 146. In some embodiments, the user provides input indicating whether the actual location of a point on an object in image 104 is included in the set of possible locations represented by line 160. As shown in diagram 100E, the actual location of a point on an object in image 104 is included in the set of possible locations represented by line 160. Figure 1C The error represented by gap 142 in has been corrected based on updated position 150, so the actual position of the point is included in the set of possible positions represented by line 160. A user may provide input confirming that the actual position of the point on the object in image 104 is included in the set of possible positions represented by line 160, and in some embodiments, identifies the actual position of the point on the object in image 104 (e.g., by selecting a point on line 160).

[0041] Once the user input has confirmed that the actual location of the point on the object in the image 104 is included in the set of possible locations represented by the line 160, and the location 150 is therefore three-dimensionally coherent, the image 102 annotated with the location 150 can be used as training data for the machine learning model. In addition, if the user identifies the actual location of the point on the object in the image 104, the image 104 annotated with the actual location of the point on the object identified by the user can also be used as training data for the machine learning model.

[0042] In an alternative embodiment, if the user input would indicate that a gap still remains between line 160 and the actual location of the point on the object in image 104, the position relative to Figure 1D and Figure 1E The process is described until user input is received confirming that the actual location of the point on the object in the image 104 is included in the set of possible locations displayed to the user.

[0043] It should be noted that, despite the Figures 1A to 1E The process described is described as occurring with certain stages being performed separately or serially, such as a user selecting a point on a first image and then separately displaying a second image with epipolar lines, but in some aspects, certain stages may be performed more in parallel (e.g., with more real-time feedback). For example, in some aspects, both the first image 102 and the second image 104 may be displayed simultaneously. In addition, when a user provides input corresponding to a location on the first image 102, a corresponding epipolar line may be displayed on the second image 104. As the user moves the input location on the first image 102 (e.g., by moving a finger on a touch screen, clicking and dragging, etc.), the epipolar line may be updated on the second image 104. Thus, the user may keep moving the input location on the first image while viewing the updated epipolar lines on the second image 104 until a suitable input location on the first image 102 is determined.

[0044] As follows relative to Figure 2 In more detail, using relative Figures 1A to 1E The training data generated by the guided feedback loop can be used to train a machine learning model to determine the three-dimensional coherent positions of points on an object in a given image.

[0045] Example Model Training

[0046] Figure 2 Depicted are examples related to training of a machine learning model for three-dimensional coherent image detection according to embodiments of the present disclosure.

[0047] The first annotated image 202 (e.g., Figure 1E 160 annotated with the actual location of the point on the object in the image) and in some embodiments a second annotated image 204 (e.g., annotated with the actual location of the user's identification of the point on the object in the image) Figure 1E The model training data 210 may include data describing one or more images (e.g., describing features of the image and / or actual data of the image) associated with a label indicating the location of a particular point of interest in the image (e.g., the location of a fingertip on a hand in the image). The location may be represented, for example, by XY coordinates relative to the image (e.g., relative to the lower left corner of the image or another origin).

[0048] The model training data 210 is used to train the machine learning model 220. In some embodiments, the training of the machine learning model 220 can be a supervised learning process that involves providing training inputs (e.g., features describing an image) as inputs to the model. The model processes the training inputs and outputs a position relative to the training inputs (e.g., XY coordinates relative to the image). The output is compared to a label associated with the training input (e.g., a known position of a point of interest) to determine the accuracy of the model, and the model is iteratively adjusted until one or more conditions are met. The machine learning model 220 may, for example, include a neural network. A neural network typically includes a collection of connected units or nodes referred to as artificial neurons. The operation of a neural network can be modeled as an iterative process. Each node has a specific value associated with it. In each iteration, each node updates its value based on the values ​​of other nodes, and the update operation typically includes a matrix-vector multiplication. The update algorithm reflects the impact of each node on other nodes in the network.

[0049] Once machine learning model 220 is trained, input may be provided to machine learning model 220 based on image 222, and machine learning model 220 outputs locations 224 within image 222, such as locations indicating points on objects in image 222 (e.g., fingertips on a hand in image 222). Training machine learning model 220 based on three-dimensionally coherent training data as described herein ensures that locations 224 output by machine learning model 220 will be three-dimensionally coherent, and therefore more accurate than locations that may be output by a model trained using training data that is not three-dimensionally coherent.

[0050] Example Method

[0051] Figure 3 Depicted are example operations 300 related to three-dimensional coherent image detection according to embodiments of the present disclosure. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, these operations are performed in accordance with the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations. In some aspects, Figure 4 The processing system 400 may perform the operation 300 .

[0052] Operations 300 begin at step 305 by receiving a first image of an object from a first perspective.

[0053] Operation 300 continues at step 310 by receiving a second image of the object from a second perspective.

[0054] Operation 300 continues at step 315 by receiving user input identifying a location in the first image that corresponds to a point on the object.

[0055] Operation 300 continues at step 320 by displaying a list of possible locations in the second image corresponding to points on the object based on the locations in the first image.

[0056] Some embodiments also include displaying, via a user interface, the series of possible positions as lines relative to the second image.

[0057] Operation 300 continues at step 325 by receiving an updated user input that identifies an updated position in the first image corresponding to the point on the object based on a series of possible positions that do not include the point on the object in the second image. In some embodiments, the additional user input includes an indication that the series of possible positions does not include the point on the object. For example, some embodiments also include an updated user input requesting identification of an updated position in the first image corresponding to the point on the object.

[0058] Operation 300 continues at step 330 by displaying an updated list of possible locations in the second image corresponding to points on the object based on the updated locations in the first image.

[0059] Operations 300 continue at step 335 by generating training data for the machine learning model based on the updated user input.

[0060] In some embodiments, the additional user input includes an updated series of possible locations including indications of points on the object, and the training data includes the first image labeled with the updated locations.

[0061] In some embodiments, the additional user input includes selecting a given location from an updated series of possible locations, and the training data includes a first image labeled with the updated location and a second image labeled with the given location.

[0062] Some embodiments include determining a series of possible locations in the second image corresponding to points on the object based on the locations in the first image and the epipolar constraints. For example, the series of possible locations may include a line. In some embodiments, the line is determined based on the locations in the first image and the optical centers of the respective cameras that captured the first image and the second image.

[0063] Example Processing System

[0064] Figure 4 Depicted are devices that include devices that are operable, configured, or adapted to perform operations for the techniques disclosed herein, such as with respect to Figure 3 An example of a processing system 400 of various components of operation is depicted and described.

[0065] Processing system 400 includes a central processing unit (CPU) 405, which in some examples may be a multi-core CPU 405. Instructions executed at CPU 405 may be loaded, for example, from a program memory 460 associated with CPU 405, or may be loaded from a memory 460 partition.

[0066] Processing system 400 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 410, a digital signal processor (DSP) 415, a neural processing unit (NPU) 420, a multimedia processing unit 425, and wireless connectivity 430 components.

[0067] The NPU 420 is, for example, typically a dedicated circuit configured to implement all necessary control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, etc. The NPU 420 is sometimes alternatively referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0068] NPU 420 may be configured, for example, to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other tasks. In some examples, multiple NPUs 420 may be instantiated on a single chip such as a system on a chip (SoC), while in other examples, multiple NPUs may be part of a dedicated machine learning accelerator device.

[0069] NPU 420 can be optimized for training or inference, or in some cases configured to balance performance between the two. For NPU 420 capable of performing both training and inference, the two tasks can generally still be performed independently.

[0070] The NPU 420 designed to accelerate training is generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing data set (usually labeled or tagged), iterating on the data set, and then adjusting model parameters 484 (such as weights and biases) to improve model performance. Typically, optimization based on error predictions involves passing back through the layers of the model and determining the gradient to reduce the prediction error.

[0071] An NPU 420 designed to accelerate inference is generally configured to operate on a complete model. Thus, such an NPU 420 can be configured to input a new piece of data and quickly process it through a trained model to generate a model output (e.g., inference).

[0072] In some aspects, NPU 420 may be implemented as part of one or more of CPU 405 , GPU 410 , and / or DSP 415 .

[0073] NPU 420 is a microprocessor dedicated to accelerating machine learning algorithms. For example, NPU 420 can operate predictive models such as artificial neural networks (ANN) or random forests (RF). In some cases, NPU 420 is designed to make it unsuitable for general-purpose computing such as performed by CPU 405. Additionally or alternatively, software support for NPU 420 may not be developed for general-purpose computing.

[0074] ANN is a hardware or software component that includes multiple connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes. In some cases, the signal between nodes includes real numbers, and the output of each node is calculated by a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how to process and send signals. During the training process, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing the loss function that corresponds in some way to the difference between the current result and the target result). The weight of the edge increases or decreases the strength of the signal sent between the nodes. In some cases, the nodes have a threshold below which no signal is sent at all. In some examples, the nodes are clustered into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0075] Convolutional neural networks (CNNs) are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs can enable processing of digital images with minimal preprocessing. CNNs can be characterized by using convolution (or cross-correlation) hidden layers. These layers apply convolution operations to the input before signaling the results to the next layer. Each convolution node can process data in a limited input domain (i.e., receptive domain). During the forward pass of a CNN, the filters at each layer can be convolved across the input volume, thereby calculating the dot product between the filter and the input. During the training process, the filters can be modified so that they activate when specific features within the input are detected.

[0076] Supervised learning is one of the three basic machine learning paradigms alongside unsupervised learning and reinforcement learning. Supervised learning is a machine learning technique based on learning a function that maps inputs to outputs based on example input-output pairs. Supervised learning generates a function for predicting labeled data based on labeled training data that includes a set of training examples. In some cases, each example is a pair that includes an input object (usually a vector) and a desired output value (i.e., a single value or output vector). The supervised learning algorithm analyzes the training data and generates an inference function that can be used to map new examples. In some cases, learning generates a function that correctly determines the class label of an unseen instance. In other words, the learning algorithm generalizes from the training data to unseen examples.

[0077] The term "loss function" refers to a function that affects how a machine learning model is trained in a supervised learning model. Specifically, during each training iteration, the output of the model is compared to the known annotation information in the training data. The loss function provides a value for how close the predicted annotation data is to the actual annotation data. After calculating the loss function, the parameters of the model are updated accordingly, and a new set of predictions are made during the next iteration.

[0078] In some aspects, the wireless connectivity 430 components may include, for example, subcomponents for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity 430 processing components are further connected to one or more antennas 435.

[0079] The processing system 400 may also include one or more sensor processing units associated with any manner of sensors, one or more image signal processors (ISP 445) associated with any manner of image sensors, and / or a navigation 450 processor, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0080] The processing system 400 may also include one or more input and / or output devices, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, speakers, microphones, and the like.

[0081] In some examples, one or more of the processors of processing system 400 may be based on the ARM or RISC-V instruction set.

[0082] The processing system 400 also includes a memory 460, which represents one or more static and / or dynamic memories, such as a dynamic random access memory 460, a flash-based static memory 460, etc. In this example, the memory 460 includes computer-executable components that can be executed by one or more of the aforementioned components of the processing system 400.

[0083] Examples of memory 460 include random access memory (RAM), read-only memory (ROM), or hard disk. Examples of memory 460 include solid-state memory and hard disk drive. In some examples, memory 460 is used to store computer-readable, computer-executable software containing instructions, which, when executed, cause the processor to perform various functions described herein. In some cases, among other things, memory 460 includes a basic input / output system (BIOS), which controls basic hardware or software operations, such as interactions with peripheral components or devices. In some cases, a memory controller operates a memory cell. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells in memory 460 store information in the form of logical states.

[0084] Specifically, in this example, memory 460 includes model parameters 484 (eg, weights, biases, and other machine learning model parameters 484). One or more of the depicted components, as well as other components not depicted, may be configured to perform various aspects of the methods described herein.

[0085] Generally, processing system 400 and / or its components may be configured to perform the methods described herein.

[0086] It is worth noting that in other aspects, aspects of the processing system 400 may be omitted, such as where the processing system 400 is a server computer, etc. For example, in other aspects, the multimedia component 425, the wireless connectivity 430, the sensor 440, the ISP 445, and / or the navigation 450 component may be omitted. Furthermore, aspects of the processing system 400 may be distributed.

[0087] Notice, Figure 4 This is merely one example, and alternative processing systems 400 having more, fewer, and / or different components may be used in other examples.

[0088] In one aspect, processing system 400 includes CPU 405 , GPU 410 , DSP 415 , NPU 420 , multimedia 425 , wireless connectivity 430 , antenna 435 , sensor 440 , ISP 445 , navigation 450 , input / output 455 , and memory 460 .

[0089] In some aspects, the sensor 440 may include an optical instrument (e.g., an image sensor, a camera, etc.) for recording or capturing images, which may be stored locally, sent to another location, etc. For example, an image sensor may capture visual information using one or more photosensitive elements that may be tuned for sensitivity to the visible spectrum of electromagnetic radiation. The resolution of such visual information may be measured in pixels, where each pixel may relate to an independent piece of captured information. In some cases, each pixel may therefore correspond to, for example, a component of a two-dimensional (2D) Fourier transform of an image. Computational methods may use pixel information to reconstruct an image captured by the device. In a camera, an image sensor may convert light incident on a camera lens into an analog or digital signal. The electronic device may then display the image on a display panel based on the digital signal. Image sensors are typically mounted on electronic devices such as smart phones, tablet personal computers (PCs), laptop PCs, and wearable devices.

[0090] In some aspects, sensor 440 may include a direct depth sensing sensor, such as radar, LIDAR, and other depth sensing sensors, as described herein.

[0091] Input / output 455 (e.g., an I / O controller) can manage input and output signals for a device. Input / output 455 can also manage peripheral devices that are not integrated into the device. In some cases, input / output 455 can represent a physical connection or port to an external peripheral device. In some cases, input / output 455 can utilize an operating system. In other cases, input / output 455 can represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, input / output 455 can be implemented as a part of a processor (e.g., CPU 405). In some cases, a user can interact with a device via input / output 455 or via a hardware component controlled by input / output 455.

[0092] In one aspect, memory 460 includes a training data generation component 465 , a training component 475 , a position output component 482 , and model parameters 484 .

[0093] According to some aspects, the training data generation component 465 performs operations described herein related to a guided feedback loop for generating three-dimensional coherent training data for an image detection machine learning model. For example, the training data generation component 465 may perform Figure 3 Operation 300.

[0094] According to some aspects, training component 475 trains a machine learning model using training data generated by training data generation component 465 (such as using supervised learning techniques).

[0095] According to some aspects, position output component 482 generates inferences, such as the location of a point on an object in an image. In some examples, position output component 482 can utilize the information generated by training data generation component 465 (such as described above with respect to Figure 3 The operation 300) generates training data to train a machine learning model (such as Figure 2 For example, Figure 2 The machine learning model 220 can be run on the system 400.

[0096] In some examples, such as where processing system 400 is a server computer, certain aspects, such as wireless connectivity component 430, antenna 435, multimedia component 425, navigation component 450, and sensor 440 may be omitted.

[0097] It is worth noting that Figure 4 Examples are used only, and many other examples and configurations of processing system 400 are possible, including implementations involving multiple computing devices.

[0098] Sample Clauses

[0099] Specific implementation examples are described in the following numbered clauses:

[0100] Item 1: A method comprising: receiving a first image of an object from a first perspective; receiving a second image of the object from a second perspective; receiving user input identifying a location in the first image corresponding to a point on the object; displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receiving updated user input, the updated user input identifying an updated location in the first image corresponding to the point on the object based on the series of possible locations not including the point on the object in the second image; displaying an updated series of possible locations in the second image corresponding to the point on the object based on the updated location in the first image; and generating training data for a machine learning model based on the updated user input.

[0101] Clause 2: The method of clause 1, wherein the training data comprises the first image labeled with the updated location.

[0102] Clause 3: The method of any of clauses 1 to 2, further comprising receiving additional user input indicating that the updated range of possible positions includes the point on the object.

[0103] Clause 4: A method according to any one of clauses 1 to 3, further comprising receiving additional user input for selecting a given location from the updated series of possible locations, and wherein the training data comprises the first image marked with the updated location and the second image marked with the given location.

[0104] Clause 5: A method according to any one of clauses 1 to 4, further comprising determining the series of possible positions in the second image corresponding to the point on the object based on the position in the first image and an epipolar constraint.

[0105] Clause 6: A method according to any one of clauses 1 to 5, wherein the series of possible positions comprises a line.

[0106] Clause 7: The method of clause 6, wherein the line is determined based on the position in the first image and an optical center of respective cameras that captured the first image and the second image.

[0107] Clause 8: The method of any one of clauses 1 to 7, further comprising receiving additional user input indicating that the series of possible positions does not include the point on the object in the second image.

[0108] Item 9: A processing system, the processing system comprising: a memory, the memory comprising computer-executable instructions; one or more processors, the one or more processors configured to execute the computer-executable instructions and cause the processing system to: receive a first image of an object from a first perspective; receive a second image of the object from a second perspective; receive user input identifying a location in the first image corresponding to a point on the object; display a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receive updated user input, the updated user input identifying an updated location in the first image corresponding to the point on the object based on the series of possible locations not including the point on the object in the second image; display an updated series of possible locations in the second image corresponding to the point on the object based on the updated location in the first image; and generate training data for a machine learning model based on the updated user input.

[0109] Clause 10: The processing system of clause 9, wherein the training data comprises the first image labeled with the updated location.

[0110] Clause 11: The processing system of any of clauses 9 to 10, wherein the one or more processors are further configured to cause the processing system to receive additional user input indicating that the updated series of possible locations includes the point on the object.

[0111] Clause 12: A processing system according to any one of clauses 9 to 11, wherein the one or more processors are further configured to cause the processing system to receive additional user input selecting a given position from the updated series of possible positions, and wherein the training data includes the first image marked with the updated position and the second image marked with the given position.

[0112] Clause 13: A processing system according to any one of clauses 9 to 12, wherein the one or more processors are further configured to enable the processing system to determine the series of possible positions in the second image corresponding to the points on the object based on the positions in the first image and the epipolar constraints.

[0113] Clause 14: The processing system of any one of clauses 9 to 13, wherein the series of possible positions comprises a line.

[0114] Clause 15: The processing system of clause 14, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

[0115] Clause 16: The processing system of any of clauses 9 to 15, wherein the one or more processors are further configured to receive additional user input indicating that the series of possible positions does not include the point on the object in the second image.

[0116] Item 17: A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to: receive a first image of an object from a first perspective; receive a second image of the object from a second perspective; receive user input identifying a location in the first image corresponding to a point on the object; display a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receive updated user input, the updated user input identifying an updated location in the first image corresponding to the point on the object based on the series of possible locations excluding the point on the object in the second image; display an updated series of possible locations in the second image corresponding to the point on the object based on the updated location in the first image; and generate training data for a machine learning model based on the updated user input.

[0117] Clause 18: The non-transitory computer-readable medium of clause 17, wherein the training data comprises the first image marked with the updated location.

[0118] Clause 19: A non-transitory computer-readable medium as described in any of clauses 17 to 18, wherein the instructions, when executed by the one or more processors, further cause the computing system to receive additional user input indicating that the updated series of possible locations includes the point on the object.

[0119] Clause 20: A non-transitory computer-readable medium according to any one of clauses 17 to 19, wherein the instructions, when executed by the one or more processors, further cause the computing system to receive additional user input selecting a given position from the updated series of possible positions, and wherein the training data includes the first image marked with the updated position and the second image marked with the given position.

[0120] Clause 21: A non-transitory computer-readable medium according to any one of clauses 17 to 20, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine the series of possible positions in the second image corresponding to the point on the object based on the position in the first image and the epipolar constraint.

[0121] Clause 22: The non-transitory computer readable medium of any one of clauses 17 to 21, wherein the series of possible positions comprises a line.

[0122] Clause 23: The non-transitory computer readable medium of Clause 22, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

[0123] Item 24: A device comprising: a component for receiving a first image of an object from a first perspective; a component for receiving a second image of the object from a second perspective; a component for receiving user input identifying a position in the first image corresponding to a point on the object; a component for displaying a series of possible positions in the second image corresponding to the point on the object based on the position in the first image; a component for receiving updated user input, the updated user input identifying an updated position in the first image corresponding to the point on the object based on the series of possible positions excluding the point on the object in the second image; a component for displaying an updated series of possible positions in the second image corresponding to the point on the object based on the updated position in the first image; and a component for generating training data for a machine learning model based on the updated user input.

[0124] Clause 25: The apparatus of clause 24, wherein the training data comprises the first image labeled with the updated location.

[0125] Clause 26: Apparatus according to any of clauses 24 to 25, further comprising means for receiving additional user input indicating that the updated range of possible positions includes the point on the object.

[0126] Clause 27: An apparatus according to any one of clauses 24 to 26, further comprising a component for receiving additional user input for selecting a given position from the updated series of possible positions, and wherein the training data comprises the first image marked with the updated position and the second image marked with the given position.

[0127] Clause 28: Apparatus according to any of clauses 24 to 27, further comprising means for determining the range of possible positions in the second image corresponding to the point on the object based on the position in the first image and an epipolar constraint.

[0128] Clause 29: Apparatus according to any of clauses 24 to 28, wherein the series of possible positions comprises a line.

[0129] Clause 30: The apparatus of clause 29, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

[0130] Additional considerations

[0131] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limited to the scope, applicability or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Each example may ignore, replace or add each procedure or component as appropriate. For example, the described method may be performed in a different order than described, and various steps may be added, omitted or combined. In addition, the features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality that are supplemented or substituted for the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0132] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0133] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0134] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Moreover, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. In addition, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0135] The method disclosed herein includes one or more steps or actions for implementing the method. The steps and / or actions of the method can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any appropriate component capable of performing the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors. Typically, in the case of operations illustrated in the accompanying drawings, those operations may have corresponding corresponding components with similar numbers plus functional components.

[0136] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless otherwise specified, the reference to the singular element is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. No claim element should be interpreted according to the provisions of 35 U.S.C. § 112 (f), unless the phrase "parts for..." is used to expressly record the element, or in the case of a method claim, the phrase "step for..." is used to record the element. All structural and functional equivalents of the elements of the various aspects described throughout the present disclosure that are known or will be known later to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recorded in the claims.

Claims

1. A method, comprising: receiving a first image of the object from a first perspective; receiving a second image of the object from a second perspective; receiving user input identifying a location in the first image corresponding to a point on the object; displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receiving an updated user input identifying an updated position in the first image corresponding to the point on the object based on the range of possible positions not including the point on the object in the second image; displaying a series of possible locations of an update in the second image corresponding to the point on the object based on the location of the update in the first image; as well as Training data for a machine learning model is generated based on the updated user input. 2 . The method of claim 1 , wherein the training data comprises the first image labeled with the updated location. 3 . The method of claim 1 , further comprising receiving additional user input indicating that the updated series of possible locations includes the point on the object.

4. The method of claim 1 further comprising receiving additional user input selecting a given location from the updated series of possible locations, and wherein the training data comprises the first image labeled with the updated location and the second image labeled with the given location. 5 . The method of claim 1 , further comprising determining the series of possible locations in the second image corresponding to the point on the object based on the location in the first image and an epipolar constraint. The method of claim 1 , wherein the series of possible positions comprises a line. 7 . The method of claim 6 , wherein the line is determined based on the position in the first image and an optical center of respective cameras that captured the first image and the second image. 8 . The method of claim 1 , further comprising receiving additional user input indicating that the series of possible locations does not include the point on the object in the second image.

9. A processing system, comprising: a memory including computer executable instructions; one or more processors configured to execute the computer-executable instructions and cause the processing system to: receiving a first image of the object from a first perspective; receiving a second image of the object from a second perspective; receiving user input identifying a location in the first image corresponding to a point on the object; displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receiving an updated user input identifying an updated position in the first image corresponding to the point on the object based on the range of possible positions not including the point on the object in the second image; displaying a series of possible locations of an update in the second image corresponding to the point on the object based on the location of the update in the first image; as well as Training data for a machine learning model is generated based on the updated user input.

10. The processing system of claim 9, wherein the training data comprises the first image labeled with the updated location.

11. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to receive additional user input indicating that the updated series of possible locations includes the point on the object.

12. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to receive additional user input selecting a given location from the updated series of possible locations, and wherein the training data includes the first image labeled with the updated location and the second image labeled with the given location.

13. The processing system of claim 9, wherein the one or more processors are further configured to cause the processing system to determine the series of possible positions in the second image corresponding to the point on the object based on the position in the first image and the epipolar constraint.

14. The processing system of claim 9, wherein the series of possible positions comprises a line.

15. The processing system of claim 14, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

16. The processing system of claim 9, wherein the one or more processors are further configured to receive additional user input indicating that the series of possible locations does not include the point on the object in the second image.

17. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to: receiving a first image of the object from a first perspective; receiving a second image of the object from a second perspective; receiving user input identifying a location in the first image corresponding to a point on the object; displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; receiving an updated user input identifying an updated position in the first image corresponding to the point on the object based on the range of possible positions not including the point on the object in the second image; displaying a series of possible locations of an update in the second image corresponding to the point on the object based on the location of the update in the first image; as well as Training data for a machine learning model is generated based on the updated user input.

18. The non-transitory computer-readable medium of claim 17, wherein the training data comprises the first image labeled with the updated location.

19. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the one or more processors, further cause the computing system to receive additional user input indicating that the updated series of possible locations includes the point on the object.

20. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the one or more processors, further cause the computing system to receive additional user input selecting a given location from the updated series of possible locations, and wherein the training data comprises the first image labeled with the updated location and the second image labeled with the given location.

21. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by the one or more processors, further cause the computing system to determine the series of possible locations in the second image corresponding to the points on the object based on the locations in the first image and the epipolar constraints.

22. The non-transitory computer readable medium of claim 17, wherein the series of possible positions comprises a line.

23. The non-transitory computer readable medium of claim 22, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

24. A device, comprising: means for receiving a first image of the object from a first perspective; means for receiving a second image of the object from a second viewing angle; means for receiving user input identifying a location in the first image corresponding to a point on the object; means for displaying a series of possible locations in the second image corresponding to the point on the object based on the location in the first image; means for receiving an updated user input identifying an updated position in the first image corresponding to the point on the object based on the range of possible positions not including the point on the object in the second image; means for displaying an updated series of possible positions in the second image corresponding to the point on the object based on the updated position in the first image; and Means for generating training data for a machine learning model based on the updated user input.

25. The apparatus of claim 24, wherein the training data comprises the first image labeled with the updated location.

26. The apparatus of claim 24, further comprising means for receiving additional user input indicating that the updated list of possible locations includes the point on the object.

27. The apparatus of claim 24, further comprising means for receiving additional user input selecting a given location from the updated series of possible locations, and wherein the training data comprises the first image labeled with the updated location and the second image labeled with the given location.

28. The apparatus of claim 24, further comprising means for determining the series of possible locations in the second image corresponding to the point on the object based on the location in the first image and an epipolar constraint.

29. The apparatus of claim 24, wherein the series of possible positions comprises a line.

30. The apparatus of claim 29, wherein the line is determined based on the location in the first image and an optical center of respective cameras that captured the first image and the second image.

Images