Semi-supervised keypoint-based model
A neural network training method using supervised and unsupervised loss functions on labeled and unlabeled data enhances keypoint prediction accuracy and reduces the need for extensive labeled data, facilitating efficient robotic tasks.
Patent Information
- Application Number
- JP2023506047
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-07-28
- Filing Date
- 2021-07-28
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2041-07-28
AI Technical Summary
Existing machine learning models require extensive labeled data for training, especially for tasks involving keypoint prediction, which is time-consuming and costly, particularly when dealing with dynamic or deformable objects.
A neural network training method that combines supervised and unsupervised loss functions using labeled and unlabeled data, leveraging synchronized multi-view images to reduce the need for labeled samples and enhance keypoint prediction accuracy.
The method allows for robust 3D keypoint tracking of dynamic objects with reduced labeled data, enabling efficient training and accurate keypoint detection for robotic tasks without relying on complex image reconstruction, thus reducing human effort and training time.
Smart Images

Figure 0007580571000015 
Figure 0007580571000016 
Figure 0007580571000017
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application claiming priority to U.S. Provisional Patent Application No. 63 / 057,861, filed July 28, 2020, the entire contents of which are incorporated herein by reference.
[0002] This specification relates to training machine learning models. [Background technology]
[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values of the parameters of the model.
[0004] Some machine learning models are deep models that use multiple model layers to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] Kullback, S. and Leibler, RA (1951) On Information and Sufficiency. The Annals of Mathematical Statistics, 22, 79-86 Summary of the Invention [Means for solving the problem]
[0006] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that trains a neural network to predict keypoints of unseen objects using a training dataset that includes labeled training data and unlabeled training data.
[0007] According to one aspect, a method is provided for training a neural network to predict keypoints of an object using a training dataset including labeled and unlabeled training data. The method includes receiving a training dataset including a plurality of training samples, each training sample including a set of synchronized images of one or more objects from a respective scene, each image in the set being synchronously taken by a respective camera from a different viewpoint, a subset of the set of synchronized images being labeled with ground truth keypoints, and the remaining images in the set being unlabeled. For each of one or more training samples in the plurality of training samples, a neural network is trained on the training samples by updating current values of parameters of the neural network to minimize a loss function. The loss function is a combination of a supervised loss function and an unsupervised loss function. The supervised loss function minimizes the difference between the ground truth keypoints and each predicted keypoint generated by the neural network for the labeled image according to the current values of parameters of the neural network. The unsupervised loss function minimizes the multi-view mismatch between the predicted keypoints generated by the neural network for unlabeled images taken from different viewpoints of the same scene, according to the current values of the parameters.
[0008] A keypoint may be any point of interest in an environment that is useful to detect to perform a particular task. For example, a keypoint may be a point on an object that is important for recognizing the object or the object's motion. A keypoint may be a point on the head, front legs, torso or tail of a moving animal. As another example, a keypoint may be a joint of an arm or leg of a robot. Images in a training dataset labeled with ground truth keypoints provide the neural network with initial target points to be considered as keypoints. Certain training methods significantly reduce the amount of labeled training data required to effectively train a neural network to predict keypoints. It will be appreciated that the training dataset may include training samples without any labeled images, only a set of unlabeled synchronous images (the empty set is also a subset).
[0009] The method may utilize multiple cameras synchronized to capture images of the environment from different viewpoints substantially simultaneously, or may utilize a single camera configured to capture multiple different viewpoints. The one or more cameras may be configured to capture a video sequence of the environment, and the images may correspond to frames of the video sequence.
[0010] One or more objects in the scene may be dynamic objects, for example, the objects may be in motion or may be deformable when interacted with.
[0011] The neural network may include one or more of: (i) one or more downsampling layers, (ii) one or more convolutional layers, (iii) one or more residual layers, (iv) one or more upsampling layers, (v) a spatial argmax layer, (vi) a bilinear upsampling layer, (vii) a 3D estimation layer, or (viii) a softmax layer.
[0012] The neural network is configured to receive an input image depicting a scene and process the input image to generate an output. The output of the neural network 120 includes a respective heatmap for each of a plurality of different possible keypoints of one or more objects in the scene. The heatmaps can be generated in parallel, i.e., in a single pass through the neural network. The heatmap for a given possible keypoint includes, for each of a plurality of locations in the image, e.g., for each pixel in the image, a respective score that represents the likelihood that the given possible keypoint is at that location.
[0013] The loss function may include an additional component to regularize the parameters.
[0014] The loss function can be a weighted combination of a supervised loss function, an unsupervised loss function, and an additional component.
[0015] The method may further include constructing, for each labeled image in the training sample and for each of the ground truth keypoints in the labeled image, a respective ground truth heatmap of the ground truth keypoints in the labeled image, and generating, using the neural network with the current values of the parameters, a respective predicted heatmap of the ground truth keypoints in the labeled image. That is, the respective predicted heatmap of the ground truth keypoints may be a predicted heatmap corresponding to the ground truth keypoints generated by processing the labeled image using the neural network. The method may further include calculating a Kullback-Leibler (KL) divergence of the respective ground truth heatmap and the respective predicted heatmap. The supervised loss function may be a combination of the calculated KL divergences. For example, the combination may be a combination of the calculated KL divergences for each ground truth keypoint in the labeled image and for each labeled image. The KL divergence may also be calculated for an unlabeled image in the training sample by deriving the location of the ground truth keypoints of the viewpoint to which the unlabeled image corresponds. A ground truth heatmap for this location can be constructed and the KL divergence between the ground truth heatmap and the predicted heatmap can be calculated. The KL divergence calculated from the unlabeled images can then be combined with the KL divergence calculated for the labeled images to provide a supervised loss function.
[0016] The method may further include, for each of the unlabeled images in the training sample, generating respective bootstrapped keypoints for the unlabeled image, where the bootstrapped keypoints are treated as labels for the unlabeled image. The method may further include, for each of the respective bootstrapped keypoints, constructing a respective bootstrapped heatmap for the bootstrapped keypoint, generating a respective predicted heatmap for the bootstrapped keypoint using a neural network according to the current values of the parameters, and calculating the KL divergence of the respective bootstrapped heatmaps and the respective predicted heatmaps. The unsupervised loss function may be a combination of the calculated KL divergences. For example, the neural network may be used to generate a predicted heatmap for each of the unlabeled images. These predicted heatmaps provide individual estimates for the location of the keypoints. Since each of the unlabeled images provides a different perspective of the same scene, the individual estimates may be combined to provide a final estimate of the location of the keypoints, i.e., the bootstrapped keypoints. The bootstrapped keypoints may then be used as labels. A heatmap may be constructed from the bootstrapped keypoints in a manner similar to how a "ground truth" heatmap is constructed for the ground truth keypoints. The KL divergence between the original predicted heatmap and the "ground truth" bootstrapped keypoint heatmap may be calculated. This process may be repeated for each keypoint, and the KL divergence calculated for each keypoint may be further combined to provide an unsupervised loss.
[0017] Generating each bootstrapped keypoint in the unlabeled image may include solving a weighted least squares optimization problem. For example, combining the individual estimates of the keypoints to provide the bootstrapped keypoint locations may include solving a weighted least squares optimization problem.
[0018] Each KL divergence may be further weighted by an associated importance factor in the loss function, which may be a function of the variance of the respective predicted heatmap.
[0019] The one or more objects in each scene may be (i) one or more robotic systems, or (ii) one or more objects in the environment with which the one or more robotic systems are interacting.
[0020] The training data set may be acquired by capturing images while the robotic system is executing automated scripted actions. For example, the robotic system may be configured to move in various orientations in the environment to provide image capture of the robot and the environment in different positions. In another example, the training data set may be acquired based on image capture of where a human operator is controlling the robotic system, such as providing a demonstration of a task.
[0021] The predicted keypoints generated by the neural network can be used to control a robotic system. For example, the keypoints can correspond to objects in the environment that the robotic system is to operate. The control system can issue actions to the robotic system based on the positions and relationships between the tracked keypoints and the state of the robotic system. Actions can include, for example, the position, velocity, or force / torque / acceleration of one or more joints or parts of the robotic system, or chaining higher level commands.
[0022] Controlling one or more robotic systems may include learning a control policy using reinforcement learning techniques. The predicted keypoints may be used as visual features for the robotic task instead of raw images. Thus, the predicted keypoints may form part of the observations / features of the environment for which the reinforcement learning system determines the action. The reinforcement learning system may then learn an action selection policy for performing the task according to any reinforcement learning technique. The system may select an action for the robotic system to perform based on the received observations of the environment, i.e. the predicted keypoints. The robotic system may then execute the action. The action interacts with the environment to generate new observations, which can then be used to select a new action. The environment may be a real-world environment or, in training, the environment may be a simulation of a real-world environment. Thus, the observations relate to the real-world environment and the selected actions relate to the actions to be performed by the robotic system. The action selection policy may be implemented as one or more neural networks to control the robotic system to perform a task while interacting with the real-world environment by obtaining observations from one or more sensors that sense the real-world environment, in this case synchronized camera images, generating predicted keypoints, and using the policy output to select an action that controls the robotic agent to perform the task.
[0023] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages: By using a novel self-supervised loss in combination with a supervised loss on a small number of labeled samples to train a neural network, the described training technique enables the trained neural network to provide robust tracking of 3D keypoints of dynamic objects captured in input images while reducing the amount of supervision, i.e., the amount of labeled samples included in the training data of the neural network.
[0024] Specifically, the described technique uses a small amount of image data labeled with predefined keypoints (e.g., human annotations) and self-supervises via multi-view 3D consistency using a large corpus of unlabeled but synchronized multi-view image data captured by multiple cameras from different viewpoints. The data used for training is generally cheap to acquire (e.g., RGB camera images). The scene itself can be dynamic as long as the cameras are synchronized. In this way, the trained neural network can generate keypoints for objects (even unseen objects, i.e., objects that do not appear in the images used to train the neural network) that match both the entire view and the human annotations, i.e., the labels of the labeled images. The described technique thus results in a robust detector based on the annotator's preferences, enabling robust and view-independent tracking of keypoints without the need for high-quality data such as depth images, 3D scans, or any other pre-framed models (e.g., object segmentation or foreground / background detection), thereby reducing the computational resources required to acquire high-quality data and reducing the complexity of the training system.
[0025] Moreover, the described technique does not rely on image space reconstruction, which is unsuitable for cases with rich backgrounds, and the keypoints detected by the trained neural network do not need to be tied to surface points, i.e., they can be interior objects or do not correspond to physical locations, as long as they are semantically and geometrically consistent.
[0026] Unlike unstructured visual representations, keypoints provide a human-understandable and actionable representation for robot controllers. By providing robust tracking of 3D keypoints of dynamic objects, the described technique allows robots controlled using trained neural networks to perform tasks requiring precision and generalization with little human effort. For reinforcement learning tasks that often require complex feedback and typically require a long time to train a robotic agent (specifically, when the task requires processing raw images), the keypoints generated by trained neural networks can serve as a good overview of the visual scene and can be used as visual features for robotic tasks to avoid the need to learn from raw images.
[0027] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0028] [Figure 1] FIG. 1 illustrates an exemplary New Network system. [Diagram 2] FIG. 1 is a flow diagram of an example process for determining a supervised loss function. [Diagram 3] FIG. 1 is a flow diagram of an example process for determining an unsupervised loss function. [Figure 4]FIG. 1 is a flow diagram of an example process for training a keypoint prediction neural network. [Diagram 5] FIG. 1 illustrates an exemplary architecture of a keypoint prediction neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] Like reference numbers and designations in the various drawings indicate like elements.
[0030] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that trains a keypoint prediction neural network to predict keypoints of unseen objects using a training dataset that includes labeled training data and unlabeled training data.
[0031] In some implementations, the object can be a dynamic object, such as a person, an animal, a vehicle, or a robotic agent. A key point of an object is a point on the object, e.g., a point that is important for recognizing the object or the motion of the object. For example, a key point may be a point on the head, front legs, torso, or tail of a moving animal. As another example, a key point may be a joint of an arm or leg of a robot. In some other implementations, the object may be a static object in an environment, e.g., an environment with which a robotic agent is interacting.
[0032] In some implementations, the keypoints predicted by the trained neural network can be used to control a robotic system to perform a robotic task, for example, by learning a control policy using reinforcement learning techniques. In these implementations, the predicted keypoints can serve as a good overview of the visual scene and can be used as visual features for the robotic task to avoid the need to learn features from raw images, thus reducing the time to train the robotic agent to perform the task. For example, given a task that requires the robotic agent to insert an audio plug into a socket, the trained neural network can be used to predict keypoints that are the plug tip, plug base, and socket location. As another example, given a task that requires the robotic agent to navigate an environment (e.g., an autonomous vehicle navigating a city street to reach a destination), the trained neural network can be used to predict keypoints of pedestrians and / or other vehicles on the city street so that the autonomous vehicle can avoid collisions with pedestrians and other vehicles and safely reach the destination. These predicted keypoints enable the robotic agent to navigate the surrounding environment and quickly complete the task with high accuracy.
[0033] An exemplary New Network system is shown in Figure 1. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.
[0034] The system 100 includes a training engine 110 configured to train a keypoint prediction neural network 120. The training engine 110 may be implemented as one or more software modules or components installed on one or more computers at one or more locations.
[0035] The keypoint prediction neural network 120 has a number of parameters. The system 100 may initialize the initial values of the parameters of the network 120 to, for example, zero or to random values sampled from an appropriate distribution.
[0036] In general, the neural network 120 is configured to receive an input image depicting a scene and process the input image to generate an output. The output of the neural network 120 includes a respective heatmap for each of a plurality of different possible keypoints of one or more objects in the scene. The heatmaps can be generated in parallel, i.e., in a single pass through the neural network. The heatmap for a given possible keypoint includes, for each of a plurality of locations (2D locations) in the image, e.g., for each pixel in the image, a respective score that represents the likelihood that the given possible keypoint is at that location.
[0037] The neural network 120 may include one or more of: (i) one or more downsampling layers, (ii) one or more convolutional layers, (iii) one or more residual layers, (iv) one or more upsampling layers, (v) a spatial argmax layer, (vi) a bilinear upsampling layer, (vii) a 3D estimation layer, or (viii) a softmax layer.
[0038] An exemplary architecture 500 of a keypoint prediction neural network 120 is shown in Figure 5. The input to the network is a single camera image 502 depicting a scene, and the output 534 of the network includes a respective heatmap for each of multiple different possible keypoints of one or more objects in the scene.
[0039] The exemplary architecture 500 of the neural network includes (i) a first convolutional downsampling layer 504, followed by (ii) a first convolutional neural network layer 508, followed by (iii) a second convolutional downsampling layer 510, followed by (iv) a second convolutional neural network layer 512, followed by (v) a stack of three residual blocks (including block 514, followed by block 516, followed by block 518), followed by (vi) a third convolutional neural network layer 520, followed by (vii) a convolutional upsampling layer 522, followed by (viii) a fourth convolutional neural network layer 524, followed by (ix) a bilinear upsampling layer 526, followed by (x) a spatial softmax layer 528.
[0040] Each of the layers of the architecture 500 receives a respective layer input and generates a respective layer output. The first convolutional downsampling layer is configured to receive an input image 502 as its layer input. The layer output of each layer is used as an input to the next layer in the architecture 500, i.e., the next layer or output layer. Each layer of the architecture 500 generates an output from the received input according to the current values of the parameters of the neural network.
[0041] In some implementations, a layer can receive as input the output of multiple preceding layers. For example, layer 526 can receive as input both the output of layer 524 and the output of layer 504. Layer 522 can receive as input both the output of layer 520 and the output of layer 510. Layer 524 can receive as input both the output of layer 522 and the output of layer 508.
[0042] The system 100 of FIG. 1 may further include a 3D estimation layer, for example 3D estimation layer 530 configured to process the heatmaps generated by the keypoint prediction neural network for the different camera images to generate an output 532 including estimates of the 3D position of each possible keypoint of one or more objects in the scene.
[0043] Returning to FIG. 1, to train the neural network 120, the training engine 110 receives a training dataset 102 including a plurality of training samples. Each training sample includes a set of synchronized images of one or more objects from a respective scene. Each image in the set is synchronously taken by a respective camera from a different viewpoint (or viewpoint) of the same scene. The set of synchronized images in each training sample includes a subset of labeled images and a subset of unlabeled images. Each labeled image in the subset of labeled images is labeled with ground truth keypoints that represent important points on one or more objects in the scene. Each image in the subset of unlabeled images is not labeled with any ground truth keypoints. It will be appreciated that some training samples may not have any labeled images.
[0044] For each of one or more training samples of the plurality of training samples, the system 100 trains a neural network 120 on the training sample by updating current values of the parameters of the neural network 120 to minimize a loss function 116. The loss function is a combination of a supervised loss function 112 and an unsupervised loss function 114.
[0045] L supThe supervised loss function 112, denoted as , minimizes the difference between the ground truth keypoints and each predicted keypoint generated by the neural network 120 for the labeled image, according to the current values of the parameters of the neural network 120.
[0046] For example, for each labeled image in the training sample and for each of the ground truth key points in the labeled image, the training engine 110 constructs a respective ground truth heat map of the ground truth key points in the labeled image. The training engine 110 generates a respective predicted heat map of the ground truth key points in the labeled image using the neural network 120 with the current values of the parameters of the neural network 120. The training engine 110 calculates a Kullback-Leibler (KL) divergence of each ground truth heat map and each predicted heat map. The supervised loss function is a combination of the calculated KL divergences. The process for determining the supervised loss function is described in more detail with reference to FIG. 2.
[0047] L unsup The unsupervised loss function 114, denoted as , minimizes the multi-view inconsistency between respective predicted keypoints generated by the neural network 120 for unlabeled images taken from different viewpoints of the same scene according to the current values of the parameters of the neural network 120. Thus, the unsupervised loss function 114 encourages the predicted keypoints generated by the neural network 120 for one or more objects in the same scene to be consistent across unlabeled images, since the unlabeled images are taken by different cameras from different viewpoints of the same scene, and thus depict the same object.
[0048] For example, for each unlabeled image in the training sample, the training engine 110 generates bootstrapped keypoints for each unlabeled image, which are treated as labels for the unlabeled image. For each of the bootstrapped keypoints, the engine 110 builds a bootstrapped heatmap for each of the bootstrapped keypoints. The engine 110 uses the neural network 120 according to the current values of the parameters to generate a predicted heatmap for each of the bootstrapped keypoints. The engine 110 calculates the KL divergence of each bootstrapped heatmap and each predicted heatmap. The unsupervised loss function is a combination of the calculated KL divergences. The process for determining the unsupervised loss function 114 is described in more detail with reference to FIG. 3.
[0049] In some implementations, the loss function 116 may include an additional component to regularize the parameters of the neural network 120. The loss function 116 may be a weighted combination of a supervised loss function, an unsupervised loss function, and an additional component. For example, L total The loss function 116, denoted as , may take the form: L total =L sup +αL unsup +λ||θ nonbias || 2 where α, λ are weights that are hyperparameters or fixed constants, and ||θ nonbias || 2 is the L2 regularization for all unbiased parameters of the keypoint prediction neural network 120.
[0050] For each batch of one or more labeled images and one or more unlabeled images of the training sample, the training engine 110 trains the neural network 120 on the training sample by updating current values of the parameters of the neural network 120 to minimize the loss function 116 using conventional neural network training techniques, e.g., gradient descent-based training techniques. In particular, the training engine 110 may determine an estimate of the gradient of the loss function 116 and back-propagate the estimate of the gradient of the loss function 116 to determine respective update values for each of the parameters of the neural network 120.
[0051] After the training engine 110 trains the neural network 120 with one or more training examples, the system 100 obtains a trained keypoint prediction neural network 122 with the latest updated values of the parameters. In some implementations, the system 100 processes a new image using the trained keypoint prediction neural network 122 to generate keypoints for one or more objects in a scene depicted by the new image. Alternatively or additionally, the system 100 can provide the trained keypoint prediction neural network 122 to another system that uses the trained keypoint prediction neural network 122 to process a new image to generate keypoints for one or more objects in a scene depicted by the new image. In any of these cases, the generated keypoints can optionally be used to perform downstream tasks, such as one of the downstream tasks described above.
[0052] 2 is a flow diagram of an example process 200 for determining a supervised loss function for training a keypoint prediction neural network. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system such as neural network system 100 of FIG. 1 suitably programmed in accordance with this specification may perform process 200.
[0053] As mentioned above, each training sample includes a subset of labeled images and a subset of unlabeled images. Each labeled image is labeled with a number of K ground truth keypoints. Each ground truth keypoint is associated with a respective 2D position coordinate.
[0054] For each labeled image c, and for each ground truth key point k of the plurality of K ground truth key points in the labeled image c, the system performs steps 202-206 as follows.
[0055] The system builds a ground truth heatmap for each of the ground truth keypoints in the labeled image (step 202). In particular, for each ground truth keypoint k, the system calculates the 3D position coordinates x k Calculate.
[0056] The system considers a labeled image I captured by a camera c. c 3D position coordinate x on the image plane k Project the labeled image I c The image coordinates of keypoint k backprojected onto i ck ,j ck This system can be expressed as: ck ,j ckWe construct a Gaussian image centered on , which results in a ground truth heatmap H of the form ck (ij) is created. H ck (ij)∝exp((-(i ck -i) 2 -(i ck -j) 2 ) / (2σ 2 )) where i and j are the coordinates in the ground truth heatmap. The width and height of the Gaussian image match the dimensions of the input labeled image c. The width σ is a predefined hyperparameter.
[0057] The system uses the keypoint prediction neural network with the current values of the neural network's parameters to generate predicted heatmaps for each of the ground truth keypoints in the labeled image (step 204).
[0058] Specifically, let f denote the keypoint prediction neural network and θ denote the parameters of the keypoint prediction neural network. The system then computes a labelled image I as follows: c The predicted heatmap (P ck (denoted as P ck = softmax(f(l c ,θ) k Here, the softmax operation is performed for each keypoint and labeled image separately. Note that the same keypoint prediction neural network with the same parameter values is applied to each viewpoint to obtain a ground truth heatmap of all ground truth keypoints in the labeled images taken from that viewpoint. The same keypoint prediction neural network is applied to each viewpoint to provide a predicted heatmap of keypoints in general.
[0059] The system calculates the Kullback-Leibler (KL) divergence of each ground truth heatmap and each predicted heatmap (step 206). The KL divergence of each ground truth heatmap and each predicted heatmap can be expressed as follows: D KL (H ck ||softmax(f(I c ,θ) k ))
[0060] For more information on the KL divergence, see Kullback, S. and Leibler, RA (1951) On Information and Sufficiency. The Annals of Mathematical Statistics, 22, 79-86.
[0061] It will be appreciated that the above method can also include remaining unlabeled images of the training samples. k can be backprojected to the unlabeled image based on the camera calibration to give 2D positions from which a ground truth heatmap can be constructed for each unlabeled image. The KL divergence can then be computed as above and included in the supervised loss function.
[0062] The system combines the KL divergences calculated for each ground truth key point in each labeled image to determine a supervised loss function (step 208). For example, the supervised loss function may be determined as follows:
[0063]
number
[0064] In some implementations, the system may combine a weighted combination of the calculated KL divergences to determine a supervised loss function. Each KL divergence may be weighted by an associated importance factor. The associated importance factor of each KL divergence may be a function of the variance of the respective predicted heatmap.
[0065] 3 is a flow diagram of an example process 300 for determining an unsupervised loss function for training a keypoint prediction neural network. For convenience, the process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system such as the neural network system 100 of FIG. 1 suitably programmed in accordance with this specification can perform the process 300.
[0066] As explained in the description of Figure 1, each training sample contains a subset of labeled images and a subset of unlabeled images.
[0067] The system uses a subset of unlabeled images in the training samples to determine an unsupervised loss function. The unlabeled images are simultaneous images of the same scene. That is, each of the unlabeled images is taken by a respective camera from a different viewpoint (or viewpoint) of the same scene, and all of the unlabeled images are taken simultaneously by the respective cameras. The system does not have explicit labels (i.e., ground truth keypoints) for each of the unlabeled images.
[0068] Thus, for each unlabeled image in the training sample, the system generates a respective bootstrapped keypoint for the unlabeled image (step 302). In particular, the system estimates the 3D position coordinates of each of the bootstrapped keypoints of one or more objects in the scene. To estimate the 3D position coordinates of each of the bootstrapped keypoints, the system solves a weighted least squares optimization problem, which is described in more detail below. The bootstrapped keypoints are treated as labels for the unlabeled image, and thus the neural network can self-supervise from its own predictions in a 3D consistent manner.
[0069] For each bootstrapped keypoint k, the system performs steps 304-308 as follows.
[0070] For each bootstrapped keypoint k, the system builds a respective bootstrapped heatmap (step 304).
[0071] The system considers an unlabeled image I taken by a camera c. c The estimated 3D position coordinates on the image plane of
[0072]
number
[0073] Project the labeled image I c The image coordinates of keypoint k backprojected onto i ck ,j ck The system computes the bootstrapped heatmaps for each bootstrapped keypoint k.
[0074]
number
[0075] Let i be the image coordinate with width σ. ck ,j ck We construct each bootstrapped heatmap as a Gaussian image centered on
[0076]
number
[0077] where i and j are the coordinates in the bootstrapped heatmap.
[0078] The system generates predicted heatmaps for each of the bootstrapped keypoints using a keypoint prediction neural network according to the current values of the neural network's parameters (step 306).
[0079] In particular, the system computes the unlabeled image I c The predicted heatmap (P ck ) is generated as follows: P ck = softmax(f(l c ,θ) k Here, the softmax operation is performed for each keypoint and the unlabeled image separately. The sum in the softmax operation is the sum of the unlabeled image I c image pixels, i.e., the image pixels are treated as different image classes.
[0080] Returning to step 302, to estimate the 3D location coordinates of each bootstrapped keypoint, the system calculates the 3D coordinates P ck ((x mean ,y mean ) from which the orientation can be calculated via calibration. The image position is given by (xmean ,y mean ) an unlabeled image I defined by c The position is in the direction P ck (In 3D coordinates, imagine there is an imaginary line running through the camera, the orientation of that line can be referred to as the direction, and the imaginary line represents that camera's ray.)
[0081] For example, P ck To determine the mean of, the system uses a probability distribution, P ck To find the average image position in x and y, the system can use mean =ΣiΣj(jP ck [i,j]) and y mean =ΣiΣj(iP ck [i,j]), where (i,j) is the image I c These are the coordinates of each pixel in
[0082] Assuming that every keypoint appears exactly once in the scene, the bootstrapped 3D localization of keypoint k (
[0083]
number
[0084] , denoted as , may be defined as the point for which the sum of the squares of the distances from these rays to the point is minimized. This means that
[0085]
number
[0086] We define a quadratic cost in a, which can then be solved analytically. In particular, if the position of camera c is a cand the normalized direction of the ray from camera c passing through bootstrapped keypoint k is
[0087]
number
[0088] By solving the weighted least squares problem above, the system can obtain the following equation, which results in the estimated 3D position coordinates:
[0089]
number
[0090]
number
[0091] are treated as labels (i.e., ground truth keypoints) and are not backpropagated during training.
[0092] It will be appreciated that step 306 may be performed as part of step 302 .
[0093] The system computes the KL divergence of each bootstrapped heatmap and each predicted heatmap (step 308).
[0094]
number
[0095] The system combines the KL divergences calculated for each bootstrapped keypoint in each unlabeled image to determine an unsupervised loss function (step 310). For example, the unsupervised loss function may be unsup and can be determined as follows:
[0096]
number
[0097] During the ceremony,
[0098]
number
[0099] are the 3D position coordinates of the bootstrapped keypoints
[0100]
number
[0101] It is created from.
[0102] In some implementations, it may not always be desirable to incorporate all predictions from all cameras, for example due to some keypoints being out of view or occluded. In addition, it may be desirable to give higher weight to camera views that have more reliable predictions of the heatmap on the keypoints. Thus, in some implementations, the system determines an unsupervised loss function based on a weighted combination of the computed KL divergences. Each KL divergence is calculated based on the weight of each predicted heatmap P ck The KL divergence predictions can be weighted by an associated importance factor that represents the importance of the predicted heatmap (i.e., predicted heatmaps associated with camera views with more reliable predictions are more important). The associated importance factor of each KL divergence can be a function of the variance of the respective predicted heatmap. For example, the associated importance factor can be calculated as follows:
[0103]
number
[0104] In the formula, var ck is the predicted heatmap P ck is the variance of, sigm is the sigmoid function, and tanh is the hyperbolic tangent function.
[0105] 4 is a flow diagram of an exemplary process for training a keypoint prediction neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a neural network system such as neural network system 100 of FIG. 1 suitably programmed in accordance with this specification can perform process 400.
[0106] The system receives a training dataset including a plurality of training samples (step 402). Each training sample includes a set of synchronized images of one or more objects from a respective scene. Each image in the set is synchronously taken by a respective camera from a different viewpoint. A subset of the set of synchronized images is labeled with ground truth keypoints, and the remaining images in the set are unlabeled.
[0107] For each training sample, the system trains the neural network on the training sample by updating the current values of the neural network's parameters to minimize a loss function (step 404). The loss function is a combination of a supervised loss function and an unsupervised loss function. The supervised loss function minimizes the difference between the ground truth keypoints and each predicted keypoint generated by the neural network for the labeled image according to the current values of the neural network's parameters. The unsupervised loss function minimizes the multi-view mismatch between each predicted keypoint generated by the neural network for the unlabeled images taken from different viewpoints of the same scene according to the current values of the parameters. In some implementations, the loss function includes an additional component for normalizing the parameters. In some implementations, the loss function is a weighted combination of the supervised loss function, the unsupervised loss function, and the additional component.
[0108] In particular, to train the neural network, for each of one or more training samples of the plurality of training samples, the system performs steps 406 and 408 as follows.
[0109] The system determines an estimate of the gradient of the loss function, step 406. The system can determine the estimate of the gradient of the loss function by using a gradient descent method, such as batch gradient descent, stochastic gradient descent, or mini-batch gradient descent.
[0110] The system backpropagates the estimate of the gradient of the loss function to determine a respective update value for each of the parameters of the neural network, step 408. For example, the system can backpropagate the estimate of the gradient of the loss function to determine a respective update value for each of the parameters using a gradient descent optimizer, such as an Adaptive Moment Estimation (Adam) optimizer or an RMSprop optimizer.
[0111] The system may repeat the above training process until one or more criteria are met, for example, until a desired performance of the neural network (e.g., a predetermined level of accuracy) is achieved, until a computational budget for training is reached, or until the neural network has been trained on a predetermined number of training samples.
[0112] After training, the system obtains a trained keypoint prediction neural network with the latest updated values of the parameters. In some implementations, the system processes a new image using the trained keypoint prediction neural network to generate predicted keypoints for one or more objects in a scene depicted by the new image. Alternatively or additionally, the system can provide the trained keypoint prediction neural network to another system that uses the trained keypoint prediction neural network to process a new image to generate predicted keypoints for one or more objects in a scene depicted by the new image.
[0113] This specification uses the term "configured" with respect to systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, during operation, causes the system to perform the operation or action. A computer program or programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0114] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus.
[0115] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, for example, code constituting a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.
[0116] A computer program, which may also be referred to or written as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a single file dedicated to the program in question, or in multiple coordinated files, e.g., a file that stores one or more modules, subprograms, or portions of code, or in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0117] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer or computers.
[0118] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as, for example, an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
[0119] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or on a central processing unit of other kinds. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by, or incorporated in, special-purpose logic circuitry. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer can be incorporated in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few examples.
[0120] Computer readable media suitable for storing computer program instructions and data include, by way of example only, all forms of non-volatile memory, media and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.
[0121] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as, for example, a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, for example, feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, speech, or haptic input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user, such as, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as, for example, a smartphone running a messaging application, and receiving a response message from the user in return.
[0122] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of the machine learning training or production, i.e. inference, workload.
[0123] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.
[0124] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, such as an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app, through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.
[0125] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, can be received at the server from the device.
[0126] Although the present specification contains many specific implementation details, these are not to be construed as limitations on the scope of any invention or scope that may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features are described above as acting in some combinations and are initially claimed as such, in some cases one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of the subcombination.
[0127] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence shown, or that all of the illustrated operations be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be incorporated together in a single software product or packaged in multiple software products.
[0128] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]
[0129] 100 Neural Network System 102 training dataset 110 Training Engine 112 Supervised Loss Function 114 Unsupervised Loss Function 116 Loss Function 120 Keypoint Prediction Neural Network 122 trained keypoint prediction neural networks 200 processes 300 processes 400 processes 500 Architecture 502 Camera Images 504 a first convolutional downsampling layer; 508 First Convolutional Neural Network Layer 510 Second convolutional downsampling layer 512 Second Convolutional Neural Network Layer 514 blocks 516 blocks 518 blocks 520 Third Convolutional Neural Network Layer 522 Convolutional Upsampling Layer 524 Fourth Convolutional Neural Network Layer 526 Bilinear Upsampling Layer 528 Spatial Softmax Layer 530 3D estimation layer 534 Output
Claims
1. 1. A method for training a neural network to predict keypoints of an object using a training dataset including labeled and unlabeled training data, comprising: receiving the training dataset including a plurality of training samples, each training sample including a set of synchronized images of one or more objects from a respective scene, each image in the set being synchronously taken by a respective camera from a different viewpoint, a subset of the set of synchronized images being labeled with ground truth keypoints, and the remaining images in the set being unlabeled; For each of one or more training samples in the plurality of training samples, training the neural network on the training samples by updating current values of parameters of the neural network to minimize a loss function; Including, the loss function is a combination of a supervised loss function and an unsupervised loss function; the supervised loss function minimizes the difference between the ground truth keypoints and each predicted keypoint generated by the neural network for the labeled image according to current values of the parameters of the neural network; the unsupervised loss function minimizes multi-view mismatch between respective predicted keypoints generated by the neural network for the unlabeled images taken from different viewpoints of the same scene according to the current values of the parameters; 13. A method according to claim 12, wherein the neural network is configured to receive an input image and, for each of a plurality of possible keypoints of one or more objects in the input image, generate a respective heatmap over the possible keypoints.
2. The method of claim 1 , wherein the one or more objects in the scene are dynamic objects.
3. 3. The method of claim 1, wherein the neural network includes one or more of: (i) one or more downsampling layers; (ii) one or more convolutional layers; (iii) one or more residual layers; (iv) one or more upsampling layers; (v) a spatial argmax layer; (vi) a bilinear upsampling layer; (vii) a 3D estimation layer; or (viii) a softmax layer.
4. The method of claim 1 , wherein the neural network is configured to generate the respective heatmaps in parallel.
5. 5. The method of claim 4, wherein a heatmap over possible keypoints includes a respective score for each of a plurality of locations in the input image, the respective score representing the likelihood that the possible keypoint is at that location.
6. The method of any one of claims 1 to 5, wherein the loss function includes an additional component for regularizing the parameters.
7. The method of claim 6 , wherein the loss function is a weighted combination of the supervised loss function, the unsupervised loss function, and the additional component.
8. The method further comprises: for each labeled image in the training sample, and for each of the ground truth keypoints in the labeled image: constructing a ground truth heatmap for each of the ground truth keypoints in the labeled images; using the neural network with the current values of the parameters to generate a predicted heatmap for each of the ground truth keypoints in the labeled image; calculating a Kullback-Leibler (KL) divergence of each of the ground truth heatmaps and each of the predicted heatmaps; Including, The supervised loss function is a combination of the calculated KL divergences. The method according to any one of claims 1 to 7.
9. The method further comprises the steps of: for each of the unlabeled images in the training sample: generating bootstrapped keypoints for each of the unlabeled images, the bootstrapped keypoints being treated as labels for the unlabeled images; For each of the respective bootstrapped keypoints, constructing a bootstrapped heatmap for each of the bootstrapped keypoints; generating a predicted heatmap for each of the bootstrapped keypoints using the neural network according to the current values of the parameters; computing the KL divergence of each of the bootstrapped heatmaps and each of the predicted heatmaps; Including, The unsupervised loss function is a combination of the calculated KL divergences. The method according to any one of claims 1 to 8.
10. The method of claim 9 , wherein generating the respective bootstrapped keypoints in the unlabeled images comprises solving a weighted least squares optimization problem.
11. The method of claim 8 or 9, wherein each KL divergence is further weighted by an associated importance factor in the loss function.
12. The method of claim 11 , wherein the associated importance coefficient of each KL divergence is a function of the variance of the respective predicted heatmap.
13. 13. The method of claim 1, wherein the one or more objects in the respective scenes are (i) one or more robotic systems, or (ii) one or more objects in an environment with which the one or more robotic systems interact.
14. The method of claim 13 , wherein the predicted keypoints are used to control the one or more robotic systems.
15. The method of claim 14 , wherein controlling the one or more robotic systems includes learning a control policy using reinforcement learning techniques.
16. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method according to any one of claims 1 to 15.
17. One or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the respective method operations recited in any one of claims 1 to 15.
Citation Information
Patent Citations
Neural networks for selecting actions to be performed by a robotic agent
JP2019530105A
Room layout estimation methods and techniques
JP2020514890A
