Semi-supervised keypoint-based model

Abstract

A method of training a neural network to predict key points of an occluded object using a training dataset comprising labeled and unlabeled training data is described. The method comprises: receiving a training dataset comprising a plurality of training samples, each training sample comprising a synchronized set of images of one or more objects from a respective scene, wherein each image in the set is synchronously captured by a respective camera from a different viewpoint, and wherein a subset of the synchronized set of images is labeled with ground truth key points, and the remaining images in the set are unlabeled; and, for each of one or more of the plurality of training samples: training the neural network on the training sample by updating current values of parameters of the neural network to minimize a loss function that is a combination of a supervised loss function and an unsupervised loss function.

Description

[0001] Cross-reference to related applications

[0002] This application is a non-provisional application filed on July 28, 2020, in which U.S. Provisional Patent Application No. 63 / 057,861, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This manual pertains to training machine learning models. Background Technology

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received inputs. Some machine learning models are parametric models, generating outputs based on the received inputs and the values ​​of the model parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which uses a training dataset including labeled and unlabeled training data to train a neural network to predict key points of unseen objects.

[0007] According to the present invention, a method is provided for training a neural network to predict keypoints of objects using a training dataset comprising labeled and unlabeled training data. The method includes receiving a training dataset comprising multiple training samples, each training sample comprising a synchronized set of images of one or more objects from a corresponding scene, wherein each image in the set is simultaneously captured by a corresponding camera from different viewpoints, and wherein a subset of the synchronized image set is labeled with ground truth keypoints, and the remaining images in the set are unlabeled. For each of the multiple training samples: the neural network is trained on the training samples by updating the current values ​​of the neural network parameters to minimize a loss function. The loss function is a combination of a supervised loss function and an unsupervised loss function. The supervised loss function minimizes the difference between the ground truth keypoints and the corresponding predicted keypoints generated by the neural network for labeled images based on the current values ​​of the neural network parameters. The unsupervised loss function minimizes multi-view inconsistencies among the corresponding predicted keypoints generated by the neural network for unlabeled images captured from different viewpoints of the same scene based on the current values ​​of the parameters.

[0008] Keypoints can be any point of interest in the environment that is useful for detecting objects used to perform a specific task. For example, a keypoint can be a point on an object that is important for recognizing the object or its movement. A keypoint can be a point on the head, forepaws, body, or tail of a moving animal. As another example, a keypoint can be a joint of a robot arm or leg. Images labeled with ground truth keypoints in the training dataset provide the neural network with initial target points that will be considered as keypoints. Specific training methods greatly reduce the amount of labeled training data needed to effectively train a neural network to predict keypoints. It will be understood that the training dataset can include training samples consisting only of a set of unlabeled, synchronized images without any labeled images (the empty set is also a subset).

[0009] This method can utilize multiple synchronized cameras to capture images of the environment from different viewpoints at substantially the same time, or it can utilize a single camera configured to capture multiple different viewpoints. The one or more cameras can be configured to capture a video sequence of the environment, and the images can correspond to frames of the video sequence.

[0010] One or more objects in the scene can be dynamic objects. For example, an object can be in motion or can deform when interacted with.

[0011] A neural network may include one or more of the following: (i) one or more downsampling layers, (ii) one or more convolutional layers, (iii) one or more residual layers, (iv) one or more upsampling layers, (v) a spatial argmax layer, (vi) a bilinear upsampling layer, (vii) a 3D estimation layer, or (viii) a softmax layer.

[0012] The neural network is configured to receive an input image depicting a scene and process the input image to generate an output. The output of the neural network 120 includes a heatmap of each of a plurality of different possible keypoints of one or more objects in the scene. The heatmaps can be generated in parallel, i.e., generated in a single pass by the neural network. The heatmap of a given possible keypoint includes a corresponding score for each of a plurality of locations in the image, for example, a score for each pixel in the image, which represents the likelihood that a given possible keypoint is located at that location.

[0013] The loss function may include additional components for regularization parameters.

[0014] The loss function can be a weighted combination of supervised loss function, unsupervised loss function and additional components.

[0015] The method may further include, for each labeled image in the training samples and for each ground truth keypoint in the labeled image: constructing a corresponding ground truth heatmap for the ground truth keypoint in the labeled image and generating a corresponding predicted heatmap for the ground truth keypoint in the labeled image using a neural network with current parameter values. That is, the corresponding predicted heatmap for the ground truth keypoint can be a predicted heatmap corresponding to the ground truth keypoint generated by processing the labeled image using a neural network. The method may further include: calculating the Kullback-Leibler (KL) divergence of the corresponding ground truth heatmap and the corresponding predicted heatmap. The supervised loss function can be a combination of the calculated KL divergences. For example, this combination can be a combination of the ground truth keypoints calculated for each ground truth keypoint in the labeled image and the KL divergence calculated for each labeled image. The KL divergence of the unlabeled image in the training samples can also be calculated by deriving the location of the ground truth keypoint for the viewpoint corresponding to the unlabeled image. A ground truth heatmap can be constructed at this location, and the KL divergence between the ground truth heatmap and the predicted heatmap can be calculated. The KL divergence calculated for unlabeled images in this manner can be combined with the KL divergence calculated for labeled images to provide a supervised loss function.

[0016] The method may further include, for each of the unlabeled images in the training samples: generating a corresponding bootstrap keypoint for the unlabeled image, wherein the bootstrap keypoint is treated as a label for the unlabeled image. The method may further include, for each corresponding bootstrap keypoint: constructing a corresponding bootstrap heatmap for the bootstrap keypoint; generating a corresponding predicted heatmap for the bootstrap keypoint using a neural network based on the current values ​​of the parameters; and calculating the KL divergence between the corresponding bootstrap heatmap and the corresponding predicted heatmap. The unsupervised loss function can be a combination of the calculated KL divergences. For example, a neural network can be used to generate predicted heatmaps for each of the unlabeled images. These predicted heatmaps provide individual estimates of the keypoint locations. Since the unlabeled images each provide different viewpoints of the same scene, the individual estimates can be combined to provide a final estimate of the location of the keypoint, i.e., the bootstrap keypoint. The bootstrap keypoint can then be used as a label. The heatmap can be constructed from the bootstrap keypoints in a similar manner to how a "ground truth" heatmap is constructed for ground truth keypoints. The KL divergence between the original predicted heatmap and the "ground truth" bootstrap keypoint heatmap can be calculated. This process can be repeated for each keypoint, and the calculated KL divergence for each keypoint can be further combined to provide unsupervised loss.

[0017] Generating corresponding bootstrap keypoints in an unlabeled image can include solving a weighted least squares optimization problem. For example, a combination of individual estimates of keypoints used to provide the locations of the bootstrap keypoints can include solving a weighted least squares optimization problem.

[0018] Each KL divergence can be further weighted by an associated importance factor in the loss function. The associated importance factor for each KL divergence can be a function of the variance of the corresponding prediction heatmap.

[0019] One or more objects in the corresponding scenario may be (i) one or more robotic systems or (ii) one or more objects in the environment with which the one or more robotic systems are interacting.

[0020] Training datasets can be obtained by capturing images while the robotic system is executing automated script actions. For example, the robotic system can be configured to move in various directions within an environment to provide image captures of the robot and the environment at different locations. In another example, training datasets can be obtained based on image captures of locations where a human operator is controlling the robotic system, such as to provide a demonstration of a task.

[0021] Predicted keypoints generated by neural networks can be used to control robotic systems. For example, keypoints can correspond to objects in the environment that the robotic system needs to manipulate. The control system can issue actions to the robotic system based on the position and relationship between the tracked keypoints and the state of the robotic system. These actions can include, for example, linking positions, velocities, or forces / torques / accelerations as commands to one or more joints or parts of the robotic system or at a higher level.

[0022] Controlling the one or more robotic systems may include learning a control policy using reinforcement learning techniques. Predicted keypoints can be used as visual features for the robotic task instead of the original image. Therefore, the predicted keypoints may form part of the observations / representations of the environment on which the reinforcement learning system determines actions. The reinforcement learning system can then learn an action selection policy for performing the task, based on any reinforcement learning technique. The system can select actions to be performed by the robotic system based on received environmental observations—i.e., predicted keypoints. The robotic system can then perform these actions. These actions interact with the environment to generate new observations, which in turn can be used to select new actions. The environment can be a real-world environment, or, during training, a simulation of a real-world environment. In this way, the observations are related to the real-world environment, and the selected actions are related to the actions to be performed by the robotic system. The action selection policy, implemented as one or more neural networks, controls the robotic system to perform the task while interacting with the real-world environment by acquiring observations from one or more sensors that sense the real-world environment, in this case, acquiring synchronized camera images and generating predicted keypoints, and using the policy output to select actions to control the robotic agent to perform the task.

[0023] The subject matter described in this specification can be implemented in particular embodiments to achieve one or more of the following advantages. By training neural networks using a novel self-supervised loss combined with a supervised loss on a small number of labeled samples, the described training technique allows the trained neural network to provide robust tracking of 3D keypoints of dynamic objects captured in an input image, while reducing the amount of supervision, i.e., the amount of labeled samples included in the neural network training data.

[0024] In particular, the described technique uses a small amount of image data labeled with predetermined key points (e.g., human annotations) and a large amount of unlabeled but synchronized multi-view image data captured by multiple cameras from different viewpoints, in order to achieve multi-view processing. Figure 3 D-consistency enables self-supervision. Training data is typically inexpensive (e.g., RGB camera images). The scene itself can be dynamic, provided the cameras are synchronized. In this way, the trained neural network can generate keypoints of objects (even unseen objects, i.e., objects not present in the images used to train the neural network) that are consistent across views and consistent with human annotations, i.e., consistent with the labels on the labeled images. Therefore, the described technique can lead to robust detectors based on annotator preferences and enables robust, view-independent tracking of keypoints without requiring high-quality data such as depth images, 3D scans, or any other pre-assembled models (e.g., object segmentation or foreground / background detection), thus reducing the computational resources required to obtain high-quality data and decreasing the complexity of the training system.

[0025] Furthermore, the described technique does not rely on image space reconstruction, which becomes inappropriate in situations with rich backgrounds. Keypoints detected by the trained neural network also do not need to be bound to surface points; that is, they can be inside objects or not correspond to any physical location, as long as they are semantically and geometrically consistent.

[0026] Unlike unstructured visual representations, keypoints are human-understandable and provide an actionable representation for robot controllers. By providing robust tracking of 3D keypoints of dynamic objects, the described technique allows robots controlled by trained neural networks to perform tasks requiring precision and generalization with minimal human intervention. For reinforcement learning tasks that often require complex feedback and typically take a long time to train robot agents (especially when the task involves processing raw images), keypoints generated by trained neural networks can serve as a good generalization of the visual scene and can be used as visual features for the robot task, avoiding the need to learn from raw images.

[0027] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0028] Figure 1 An example neural network system is shown.

[0029] Figure 2 This is a flowchart of an example process for determining a supervised loss function.

[0030] Figure 3 This is a flowchart of an example process for determining an unsupervised loss function.

[0031] Figure 4 This is a flowchart of an example process for training a keypoint prediction neural network.

[0032] Figure 5 The illustration shows an example architecture of a keypoint prediction neural network.

[0033] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0034] This specification describes a system implemented as a computer program on one or more computers at one or more locations, the computer program training a keypoint prediction neural network to predict keypoints of unseen objects using a training dataset that includes labeled and unlabeled training data.

[0035] In some implementations, the object can be a dynamic object, such as a person, animal, vehicle, or robotic agent. Key points of an object are points on the object, such as points important for identifying the object or its movement. For example, key points could be points on the head, forepaws, body, or tail of a moving animal. As another example, key points could be joints of a robotic arm or leg. In some other implementations, the object can be a static object in an environment, such as the environment with which the robotic agent is interacting.

[0036] In some implementations, keypoints predicted by a trained neural network can be used to control a robotic system to perform robotic tasks, for example, by learning control strategies using reinforcement learning techniques. In these implementations, the predicted keypoints can serve as a good generalization of the visual scene and can be used as visual features for the robotic task, avoiding the need to learn features from raw images and thus reducing the time spent training the robotic agent to perform the task. For example, given a task requiring a robotic agent to insert an audio plug into a socket, a trained neural network can be used to predict keypoints as the locations of the plug tip, plug base, and socket. As another example, given a task requiring a robotic agent to navigate an environment (e.g., an autonomous vehicle navigating a street to reach a destination), a trained neural network can be used to predict keypoints of pedestrians and / or other vehicles on the street, enabling the autonomous vehicle to avoid collisions with pedestrians and other vehicles and safely reach its destination. These predicted keypoints will allow the robotic agent to navigate its surroundings and complete tasks quickly and with high accuracy.

[0037] Figure 1 An example neural network system is shown. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the following systems, components, and techniques can be implemented.

[0038] System 100 includes a training engine 110 configured to train a keypoint prediction neural network 120. The training engine 110 may be implemented as one or more software modules or components installed on one or more computers in one or more locations.

[0039] The keypoint prediction neural network 120 has multiple parameters. The system 100 can initialize the initial values ​​of the parameters of the network 120 to, for example, zero or random values ​​sampled from an appropriate distribution.

[0040] Typically, neural network 120 is configured to receive an input image depicting a scene and process the input image to generate an output. The output of neural network 120 includes a heatmap of each of a plurality of different possible keypoints of one or more objects in the scene. The heatmaps can be generated in parallel, i.e., generated in a single pass by the neural network. The heatmap of a given possible keypoint includes a corresponding score for each of a plurality of locations (2D locations) in the image—e.g., each pixel in the image—representing the likelihood that the given possible keypoint is located at that location.

[0041] The neural network 120 may include one or more of the following: (i) one or more downsampling layers, (ii) one or more convolutional layers, (iii) one or more residual layers, (iv) one or more upsampling layers, (v) a spatial argmax layer, (vi) a bilinear upsampling layer, (vii) a 3D estimation layer, or (viii) a softmax layer.

[0042] Figure 5 The diagram illustrates an example architecture 500 of a keypoint prediction neural network 120. The network's input is a single camera image 502 depicting the scene, and the network's output 534 includes a heatmap of each of a plurality of different possible keypoints of one or more objects in the scene.

[0043] The example architecture 500 of the neural network includes (i) a first convolutional downsampling layer 504, followed by (ii) a first convolutional neural network layer 508, followed by (iii) a second convolutional downsampling layer 510, followed by (iv) a second convolutional neural network layer 512, followed by (v) a stack of three residual blocks (including block 514, followed by block 516, followed by block 518), followed by (vi) a third convolutional neural network layer 520, followed by (vii) a convolutional upsampling layer 522, followed by (viii) a fourth convolutional neural network layer 524, followed by (ix) a bilinear upsampling layer 526, followed by (x) a spatial softmax layer 528.

[0044] Each layer of architecture 500 receives a corresponding layer input and generates a corresponding layer output. The first convolutional downsampling layer is configured to receive the input image 502 as its layer input. The layer output of each layer is used as the input to the next layer in architecture 500—that is, the next layer or output layer. Each layer of architecture 500 generates an output from the received input based on the current values ​​of the neural network parameters.

[0045] In some implementations, a layer can receive the outputs of multiple preceding layers as inputs. For example, layer 526 can receive both the outputs of layer 524 and layer 504 as inputs. Layer 522 can receive both the outputs of layer 520 and layer 510 as inputs. Layer 524 can receive both the outputs of layer 522 and layer 508 as inputs.

[0046] Figure 1 The system 100 may further include a 3D estimation layer, for example, a 3D estimation layer 530, which is configured to process heatmaps generated by a keypoint prediction neural network for different camera images to generate an output 532, the output 532 including an estimate of the 3D position of each possible keypoint of one or more objects in the scene.

[0047] Return to reference Figure 1To train the neural network 120, the training engine 110 receives a training dataset 102 comprising multiple training samples. Each training sample comprises a synchronized set of images from one or more objects in a corresponding scene. Each image in this set is simultaneously captured by a corresponding camera from different viewpoints (or perspectives) of the same scene. The synchronized set of images in each training sample comprises a subset of labeled images and a subset of unlabeled images. Each labeled image in the subset of labeled images is labeled with ground truth keypoints, which represent important points on one or more objects in the scene. Each image in the subset of unlabeled images is not labeled with any ground truth keypoints. It will be understood that some training samples may not have any labeled images.

[0048] For each of one or more training samples from a plurality of training samples, system 100 trains neural network 120 on the training samples by updating the current values ​​of the parameters of neural network 120 to minimize loss function 116. The loss function is a combination of supervised loss function 112 and unsupervised loss function 114.

[0049] Represented as L sup The supervised loss function 112 minimizes the difference between the ground truth keypoints and the corresponding predicted keypoints generated by the neural network 120 for the labeled image based on the current values ​​of the parameters of the neural network 120.

[0050] For example, for each labeled image and each ground truth keypoint in the labeled image from the training samples, training engine 110 constructs a corresponding ground truth heatmap for the ground truth keypoint in the labeled image. Training engine 110 generates a corresponding predicted heatmap for the ground truth keypoint in the labeled image using neural network 120 with the current values ​​of its parameters. Training engine 110 computes the Kullback-Leibler (KL) divergence between the corresponding ground truth heatmap and the corresponding predicted heatmap. The supervised loss function is a combination of the computed KL divergences. (Reference) Figure 2 The process for determining the supervised loss function is described in more detail.

[0051] Represented as L unsup The unsupervised loss function 114 minimizes the multi-view inconsistency among the corresponding predicted keypoints generated by the neural network 120 for unlabeled images taken from different viewpoints of the same scene, based on the current values ​​of the parameters of the neural network 120. Therefore, the unsupervised loss function 114 encourages the predicted keypoints generated by the neural network 120 for one or more objects in the same scene to remain consistent across unlabeled images, since the unlabeled images are taken by different cameras from different viewpoints of the same scene and thus depict the same objects.

[0052] For example, for each unlabeled image in the training samples, the training engine 110 generates corresponding bootstrap keypoints for the unlabeled image, which are treated as labels for the unlabeled image. For each corresponding bootstrap keypoint, the engine 110 constructs a corresponding bootstrap heatmap for the bootstrap keypoint. The engine 110 uses the neural network 120 to generate a corresponding predicted heatmap for the bootstrap keypoint based on the current values ​​of the parameters. The engine 110 calculates the KL divergence between the corresponding bootstrap heatmap and the corresponding predicted heatmap. The unsupervised loss function 114 is a combination of the calculated KL divergences. (Reference) Figure 3 The process for determining the unsupervised and supervised loss function 114 is described in more detail.

[0053] In some implementations, loss function 116 may include additional components to regularize the parameters of neural network 120. Loss function 116 may be a weighted combination of supervised loss function, unsupervised loss function, and additional components. For example, it may be denoted as L. total The loss function 116 can be in the following form:

[0054] L total =L sup +αL unsup +λ||θ nonbias || 2

[0055] Where α and λ are weights that are hyperparameters or fixed constants, and ||θ nonbias || 2 It is an L2 regularization of all unbiased parameters of the keypoint prediction neural network 120.

[0056] For each batch of training samples containing one or more labeled images and one or more unlabeled images, training engine 110 trains neural network 120 on the training samples by updating the current values ​​of the parameters of neural network 120 to minimize loss function 116 using conventional neural network training techniques—such as gradient descent-based training techniques. Specifically, training engine 110 may determine an estimate of the gradient of loss function 116 and backpropagate the estimate of the gradient of loss function 116 to determine the corresponding updated values ​​of the parameters of neural network 120.

[0057] After training engine 110 has trained neural network 120 on one or more training examples, system 100 obtains a trained keypoint prediction neural network 122 with final updated values ​​of its parameters. In some embodiments, system 100 uses the trained keypoint prediction neural network 122 to process new images to generate keypoints for one or more objects in a scene depicted by the new images. Alternatively or additionally, system 100 may provide the trained keypoint prediction neural network 122 to another system, which uses the trained keypoint prediction neural network 122 to process new images to generate keypoints for one or more objects in a scene depicted by the new images. In any of these cases, the generated keypoints may optionally be used to perform downstream tasks, such as one of the downstream tasks described above.

[0058] Figure 2 This is a flowchart of an example process 200 for determining a supervised loss function for training a keypoint prediction neural network. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as... Figure 1 The neural network system 100 can execute process 200.

[0059] As discussed above, each training sample consists of a subset of labeled images and a subset of unlabeled images. Each labeled image is labeled with multiple K ground truth keypoints. Each ground truth keypoint is associated with a corresponding 2D location coordinate.

[0060] For each labeled image c and for each ground truth key point k among the multiple ground truth key points in labeled image c, the system performs the following steps 202-206.

[0061] The system constructs a corresponding ground truth heatmap for the ground truth keypoints in the labeled image (step 202). Specifically, for each ground truth keypoint k, the system calculates the 3D position coordinates xk of the ground truth keypoint based on its 2D position coordinates and camera calibration.

[0062] The system will use the 3D position coordinates x k Projected onto the marked image I captured by camera c c On the image plane. Back-projected onto the marked image I c The image coordinates of the key point k in the image can be represented as i ck j ck The system is constructed based on image coordinates i. ck j ck A Gaussian image centered at σ and with width σ is generated. This creates a corresponding ground truth heatmap H of the following form. ck (ij):

[0063] H ck (ij)∝exp((-(i ck -i) 2 -(i ck -j) 2 ) / (2σ 2 ))

[0064] Here, i and j are the coordinates in the ground truth heatmap. The width and height of the Gaussian image are matched to the dimensions of the input labeled image c. The width σ is a predefined hyperparameter.

[0065] The system uses the keypoint prediction neural network with the current values ​​of the neural network parameters to generate a corresponding predicted heatmap of the ground truth keypoints in the labeled image (step 204).

[0066] Specifically, let f denote the keypoint prediction neural network, and θ denote the parameters of the keypoint prediction neural network. The system generates a labeled image I. c The corresponding predicted heatmap of the true value key point k in the image (represented as P). ck )as follows:

[0067] P ck =softmax(f(I c ,θ) k ,

[0068] Specifically, a softmax operation is performed on each keypoint and labeled image. Note that the same keypoint prediction neural network with identical parameter values ​​is applied to each viewpoint to obtain a ground truth heatmap of all ground truth keypoints in the labeled image taken from that viewpoint. Applying the same keypoint prediction neural network to each viewpoint provides an overall predicted heatmap of the keypoints.

[0069] The system calculates the Kullback-Leibler (KL) divergence between the corresponding ground truth heatmap and the corresponding predicted heatmap (step 206). The KL divergence between the corresponding ground truth heatmap and the corresponding predicted heatmap can be expressed as follows:

[0070] D KL (H ck ||softmax(f(I c ,θ) k )).

[0071] The KL divergence is described in Kullback, S. and Leibler, RA (1951) On Information and Sufficiency. The Annals of Mathematical Statistics, 22, 79-86.

[0072] It will be understood that the above method can also include the remaining unlabeled images of the training samples. The 3D location coordinates xk of the computed ground truth keypoints can be backprojected onto the unlabeled images based on camera calibration to give 2D locations from which a ground truth heatmap can be constructed for each unlabeled image. The KL divergence can then be computed as described above and included in the supervised loss function.

[0073] The system combines the KL divergence calculated for each ground truth keypoint in each labeled image to determine the supervised loss function (step 208). For example, the supervised loss function can be determined as follows:

[0074]

[0075] In some implementations, the system can combine weighted combinations of the calculated KL divergences to determine the supervised loss function. Each KL divergence can be weighted by an associated importance factor. The associated importance factor for each KL divergence can be a function of the variance of the corresponding prediction heatmap.

[0076] Figure 3 This is a flowchart of an example process 300 for determining an unsupervised loss function for training a keypoint prediction neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as... Figure 1 The neural network system 100 can execute process 300.

[0077] like Figure 1 As described in the description, each training sample includes a subset of labeled images and a subset of unlabeled images.

[0078] The system uses a subset of unlabeled images from the training samples to determine the unsupervised loss function. The unlabeled images are simultaneous images of the same scene. This means that each unlabeled image was captured by a corresponding camera from a different viewpoint (or perspective) of the same scene, and all unlabeled images were captured simultaneously by the corresponding camera. For each unlabeled image, the system does not have an explicit label (i.e., a ground truth keypoint).

[0079] Therefore, for each of the unlabeled images in the training samples, the system generates a corresponding bootstrap keypoint for the unlabeled image (step 302). Specifically, the system estimates the 3D position coordinates of each of the bootstrap keypoints for one or more objects in the scene. To estimate the 3D position coordinates of each of the bootstrap keypoints, the system solves a weighted least squares optimization problem, which will be described in more detail below. The bootstrap keypoints are treated as labels for the unlabeled images, thus allowing the neural network to self-supervise itself based on its own predictions in a 3D consistent manner.

[0080] For each bootstrap keypoint k in the corresponding bootstrap keypoints, the system executes the following steps 304-308.

[0081] The system constructs a corresponding bootstrap heatmap for the bootstrap key point k (step 304).

[0082] The system will estimate the 3D position coordinates Projected onto the unlabeled image I captured by camera c c On the image plane. Backproject the keypoint k onto the labeled image I. c The image coordinates in the image can be represented as i ck j ck The system will generate the corresponding bootstrap heatmap for the bootstrap key point k. Constructed with image coordinates i ck j ck A Gaussian image centered at a value of σ is generated. This creates the corresponding bootstrap heatmap.

[0083] Where i and j are the coordinates in the bootstrap heatmap.

[0084] The system uses the key point prediction neural network to generate corresponding prediction heatmaps for bootstrap key points based on the current values ​​of the neural network parameters (step 306).

[0085] Specifically, the system generates unlabeled image I as follows: c The corresponding prediction heatmap of the bootstrap key point k (denoted as P) ck ):

[0086] P ck =softmax(f(I c ,θ) k ,

[0087] Specifically, a softmax operation is performed for each keypoint and unlabeled image. The sum of the softmax operations spans the unlabeled image I. c Image pixels occur, meaning that image pixels are treated as different image categories.

[0088] Returning to step 302, in order to estimate the 3D position coordinates of each bootstrap keypoint, the system determines P corresponding to the image position. ck The average value (represented as (x) mean y mean From this, the orientation can be calculated via calibration. The image position is determined by (x... mean y mean Unlabeled image I defined c Position within. Direction based on P. ck A ray is created for each camera along the direction of the calculated image position (in 3D coordinates, imagine a virtual line passing through the camera; the direction of the line can be called the orientation, and the virtual line will represent the ray of that camera).

[0089] For example, in order to determine P ck The average value, the system can use P ck The expected value in the image index (i.e., image category) is determined as a probability distribution. To find the image location where the mean of x and y is found, the system can calculate x... mean =∑ i ∑ j (jP ck [i, j]) and y mean =∑ i ∑ j (iP ck [i,j]), where (i,j) represents image I. c The coordinates of each pixel in the image.

[0090] Assuming each keypoint appears exactly once in the scene, then the bootstrap keypoint k (denoted as...) The 3D position estimate of a point can be defined as the point where the sum of the squares of the distances from these rays to that point is minimized. This defines... The quadratic cost can therefore be solved analytically. Specifically, assume the position of camera c is denoted as a. c And the normalized direction of the ray emitted from camera c through the bootstrap keypoint k is represented as follows: By solving the above weighted least squares problem, the system can obtain the following equation, which will produce the estimated 3D position coordinates:

[0091]

[0092] They are treated as labels (i.e., ground truth keypoints) and are not passed through during backpropagation during training.

[0093] It will be understood that step 306 can be performed as part of step 302.

[0094] The system calculates the KL divergence of the corresponding bootstrap heatmap and the corresponding predicted heatmap as follows (step 308):

[0095]

[0096] The system combines the KL divergence calculated for each bootstrap keypoint in each unlabeled image to determine the unsupervised loss function (step 310). For example, the unsupervised loss function can be expressed as L unsup And the following determinations are made:

[0097]

[0098] in, From the 3D position coordinates of the bootstrap keypoints Created.

[0099] In some implementations, it is not always desirable to merge all predictions from all cameras, for example, due to some keypoints disappearing from the view or becoming occluded. Additionally, it may be desirable to impose higher weights on the camera views by making more confident predictions on the heatmaps at the keypoints. Therefore, in some implementations, the system determines the unsupervised loss function based on a weighted combination of computed KL divergences. Each KL divergence can be weighted by an associated importance factor, which represents each predicted heatmap P. ck The importance of the predicted heatmap (i.e., the predicted heatmap associated with the camera view that has more confident predictions) is considered more important. The associated importance factor for each KL divergence can be a function of the variance of the corresponding predicted heatmap. For example, the associated importance factor can be calculated as follows:

[0100]

[0101] Where, var ck The corresponding predicted heatmap P ck The variance of , where sigm is the sigmoid function and tanh is the hyperbolic tangent function.

[0102] Figure 4 This is a flowchart of an example process for training a keypoint prediction neural network. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as... Figure 1 The neural network system 100 can execute process 400.

[0103] The system receives a training dataset comprising multiple training samples (step 402). Each training sample comprises a set of synchronized images of one or more objects from the corresponding scene. Each image in the set is captured synchronously by a corresponding camera from a different viewpoint. A subset of the synchronized image set is labeled using ground truth keypoints, and the remaining images in the set are unlabeled.

[0104] For each training sample, the system trains the neural network on the training samples by updating the current values ​​of the neural network parameters to minimize the loss function (step 404). The loss function is a combination of a supervised loss function and an unsupervised loss function. The supervised loss function minimizes the difference between the ground truth keypoints and the corresponding predicted keypoints generated by the neural network for the labeled image based on the current values ​​of the neural network parameters. The unsupervised loss function minimizes multi-view inconsistencies among the corresponding predicted keypoints generated by the neural network for unlabeled images taken from different viewpoints of the same scene based on the current values ​​of the parameters. In some implementations, the loss function includes an additional component for regularizing the parameters. In some implementations, the loss function is a weighted combination of the supervised loss function, the unsupervised loss function, and the additional component.

[0105] Specifically, in order to train the neural network, for each of one or more training samples among a plurality of training samples, the system performs the following steps 406 and 408.

[0106] The system determines an estimate of the gradient of the loss function (step 406). The system can determine the estimate of the gradient of the loss function by using gradient descent techniques such as batch gradient descent, stochastic gradient descent, or mini-batch gradient descent.

[0107] The system backpropagates an estimate of the gradient of the loss function to determine the corresponding update value for each of the parameters in the neural network (step 408). For example, the system can use a gradient descent optimizer—such as the Adaptive Moments Estimator (Adam) optimizer or the RMSprop optimizer—to backpropagate an estimate of the gradient of the loss function to determine the corresponding update value for each of the parameters.

[0108] The system can repeat the above training process until one or more criteria are met, such as until the desired performance of the neural network is achieved (e.g., a predetermined level of accuracy), until the computational budget for training has been reached, or until the neural network has been trained on a predetermined number of training samples.

[0109] After training, the system obtains a trained keypoint prediction neural network with finally updated parameter values. In some implementations, the system uses the trained keypoint prediction neural network to process new images to generate predicted keypoints for one or more objects in a scene depicted by the new images. Alternatively or additionally, the system may provide the trained keypoint prediction neural network to another system, which uses the trained keypoint prediction neural network to process new images to generate predicted keypoints for one or more objects in a scene depicted by the new images.

[0110] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0111] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0112] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or further include a dedicated logic circuit system, such as a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0113] Computer programs can be written in any form of programming language, and can also be called or described as programs, software, software applications, applications, modules, software modules, scripts, or code. This "any form of programming language" includes compiled or interpreted languages, or declarative or procedural languages. The computer program can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program can be stored as a part of a file that holds other programs or data—for example, one or more scripts stored in a markup language document—in a single file dedicated to the program in question, or in multiple collaborating files, such as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0114] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same computer or on multiple computers.

[0115] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0116] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both, for storing data, such as disks, magneto-optical disks, or optical disks. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0117] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0118] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from a device used by the user; for example, sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device—for example, a smartphone running a messaging application—and receiving response messages in return from the user.

[0119] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0120] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0121] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component as an application server, or a frontend component as a client computer having, for example, a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium—for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0122] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the user device.

[0123] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0124] Similarly, although operations are depicted in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring the operations to be performed in the specific order shown or in sequential order, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0125] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a neural network to predict key points of unseen objects using a training dataset comprising labeled training data and unlabeled training data, the method comprising: The system receives a training dataset comprising multiple training samples, each training sample comprising a synchronized set of images from one or more objects in a corresponding scene, wherein each image in the synchronized set is simultaneously captured by a corresponding camera from a different viewpoint, and wherein a subset of the synchronized set is labeled using ground truth keypoints, and the remaining images in the synchronized set are unlabeled; and For each of one or more training samples among the plurality of training samples: The neural network is trained on the training samples by updating the current values ​​of its parameters to minimize the loss function. The loss function is a combination of a supervised loss function and an unsupervised loss function. The supervised loss function minimizes the difference between the ground truth keypoints and the corresponding predicted keypoints generated by the neural network for the labeled image based on the current values ​​of the parameters of the neural network. The unsupervised loss function minimizes multi-view inconsistencies among the corresponding predicted keypoints generated by the neural network for unlabeled images taken from different viewpoints of the same scene, based on the current values ​​of the parameters. The method is characterized by... The neural network is configured to receive an input image and generate a heatmap for each of a plurality of possible keypoints of one or more objects in the input image. The heatmap for each possible keypoint includes a score for each of a plurality of locations in the input image, the score representing the likelihood that the possible keypoint is located at the location.

2. The method according to claim 1, wherein, The one or more objects in the scenario are dynamic objects.

3. The method according to claim 1, wherein, The neural network includes one or more of the following: (i) one or more downsampling layers, (ii) one or more convolutional layers, (iii) one or more residual layers, (iv) one or more upsampling layers, (v) a spatial argmax layer, (vi) a bilinear upsampling layer, (vii) a 3D estimation layer, and (viii) a softmax layer.

4. The method according to claim 1, wherein, The neural network is configured to generate the corresponding heatmaps in parallel.

5. The method according to claim 1, wherein, The loss function includes an additional component for regularizing the parameters.

6. The method according to claim 5, wherein, The loss function is a weighted combination of the supervised loss function, the unsupervised loss function, and the additional components.

7. The method according to claim 1, wherein, The method includes, for each labeled image in the training samples and for each of the ground truth keypoints in the labeled images: Construct a ground truth heatmap for the ground truth key points in the labeled image. The neural network, using the current values ​​of the parameters, generates a corresponding predicted heatmap of the ground truth keypoints in the labeled image, and Calculate the Kullback-Leibler (KL) divergence between the corresponding true value heatmap and the corresponding predicted heatmap, and The supervised loss function is a combination of the calculated KL divergences.

8. The method according to claim 1, wherein, The method includes, for each of the unlabeled images in the training samples: For the unlabeled image, corresponding bootstrap keypoints are generated, wherein the bootstrap keypoints are considered as labels for the unlabeled image. For each of the corresponding bootstrap keypoints: Construct the corresponding bootstrap heatmaps for the aforementioned bootstrap key points. Based on the current value of the parameters, the neural network is used to generate a corresponding predicted heatmap for the bootstrap keypoints. Calculate the KL divergence of the corresponding bootstrap heatmap and the corresponding predicted heatmap, and The unsupervised loss function is a combination of the calculated KL divergences.

9. The method according to claim 8, wherein, Generating the corresponding bootstrap keypoints for the unlabeled image includes solving a weighted least squares optimization problem.

10. The method according to claim 7, wherein, Each KL divergence is further weighted by the associated importance factor in the loss function.

11. The method according to claim 10, wherein, The associated importance factor for each KL divergence is a function of the variance of the corresponding predicted heatmap.

12. The method according to any one of claims 1 to 11, wherein, The one or more objects in the corresponding scenario are (i) one or more robot systems or (ii) one or more objects in the environment with which the one or more robot systems are interacting.

13. The method according to claim 12, wherein, The predicted key points are used to control the one or more robot systems.

14. The method according to claim 13, wherein, Controlling the one or more robotic systems includes using reinforcement learning techniques to learn control strategies.

15. A system comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1 to 14.

16. A computer-readable storage medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform the operations of the method according to any one of claims 1 to 14.

Images