Detecting objects in an image by generating a sequence of tokens

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object detection using a neural network. In one aspect, one of the methods includes obtaining an input image, processing the input image with an object detection neural network to generate an output sequence including a respective token at each of a plurality of time steps, each token selected from a vocabulary of tokens including (i) a first set of tokens, each token representing a respective discrete number from a set of discretized numbers, and (ii) a second set of tokens, each token representing a respective object category from a set of object categories, and generating an object detection output for the input image from the tokens in the output sequence.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 245,783, filed September 17, 2021. The disclosure of the prior application is considered part of, and is incorporated by reference into, the disclosure of this application.

[0002] This specification relates to processing inputs using neural networks. [Background technology]

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output given a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, either another hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. [Non-patent document 2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly et al., An image is worth 16x16 words: Transformers for image recognition at scale, International Conference on Learning Representations, 2020 [Non-patent document 3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, End-to-end object detection with transformers, European Conference on Computer Vision, pages 213–229, Springer, 2020. [Non-patent document 4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019 [Non-Patent Document 5] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, Improving language understanding by generative pre-training, 2018 [Non-patent document 6] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph, Simple copy-paste is a strong data augmentation method for instance segmentation, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2918–2928, 2021. [Non-Patent Document 7] Andrew G Howard, Some improvements on deep convolutional neural network based image classification, arXiv preprint arXiv:1312.5402, 2013 Summary of the Invention [Means for solving the problem]

[0005] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that detects objects in an input image by using a neural network to process the image and generate an output sequence of tokens.

[0006] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0007] Existing object detection methods need to be highly customized and have complex architectures, making them difficult to incorporate into larger systems. In contrast, the described technique uses an object detection neural network that generates a sequence of discrete tokens corresponding to an object description. The object detection neural network has a simple architecture and can therefore be easily incorporated into perception systems and easily extended to different domains or applications. Furthermore, despite its simple architecture, because the described technique generates a sequence of discrete tokens corresponding to an object description (bounding box and class label), the object detection neural network can achieve performance comparable to or exceeding that of much more complex systems highly designed for specific object detection tasks.

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0009] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram of an exemplary object detection system. [Figure 2] 1 is a flow diagram of an example process for detecting an object in an input image. [Figure 3] FIG. 1 illustrates an example of an object detection output generated using an object detection neural network. [Figure 4] 1 is a flow diagram of an example process for training an object detection neural network. [Figure 5] FIG. 1 illustrates training of an object detection neural network on training examples. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference numbers and designations in the various drawings indicate like elements.

[0012] 1 is a diagram of an exemplary object detection system 100. Object detection system 100 is an example of a system in which the systems, components, and techniques described below may be implemented, implemented as a computer program on one or more computers at one or more locations.

[0013] The object detection system 100 is a system that receives an input image 102, detects objects in the input image 102 by processing the image 102 using an object detection neural network 110, and generates an object detection output 150 related to the input image 102.

[0014] The object detection output 150 identifies one or more bounding boxes in the input image 102, each of which corresponds to a detected object, i.e., is predicted to contain a representation of the detected object, and for each of the bounding boxes, identifies an object category from a set of object categories to which the detected object in the bounding box belongs.

[0015] More specifically, the system 100 acquires an input image 102 .

[0016] The system 100 processes the input image 102, i.e., processes the intensity values ​​of the pixels of the input image 102, using an object detection neural network 110 to generate an output sequence 112 that includes a plurality of tokens.

[0017] Each token in the sequence is selected from a vocabulary of tokens that includes (i) a first set of tokens, each representing a respective discrete number from a set of discretized numbers, and (ii) a second set of tokens, each representing a respective object category from a set of object categories. The tokens in the vocabulary may be represented in any suitable manner, for example, as integers or other alphanumeric symbols that are distinct from one another.

[0018] More specifically, the system discretizes the numbers used to represent the coordinates of any given point in the input image 102 into a number of bins, each corresponding to a respective one of the tokens in the first set of tokens. As a specific example, the system may discretize a range of values ​​between zero and the height or width of the image in pixels into a fixed number of equally spaced bins, such that each bin corresponds to a different subset of pixel indices that may be used to represent the coordinates of a point (in pixels) in the image. Thus, if the input image is 600 pixels by 600 pixels and there are 600 bins, each bin corresponds to a different pixel index from 1 to 600. If the input image is 600 by 600 and there are 300 bins, each bin corresponds to a different set of two pixel indices from 1 to 600.

[0019] Thus, each token in the first set of tokens represents a different bin in a discretization of possible coordinate values ​​for pixels in the image and may be mapped to a different quantized coordinate value, e.g., a representative value for the bin represented by the token. For example, the representative value may be the average of the bin's endpoints or one of the bin's two endpoints. This quantization scheme for coordinates allows the system to use a relatively small vocabulary to represent possible pixel coordinates while maintaining high accuracy.

[0020] For the second set of tokens, the system can assign a different unique token to each object category in the set.

[0021] Thus, as a specific example, if there are 600 bins in the quantization scheme and 100 object categories, and tokens are represented as integers, the vocabulary may include tokens 1-600 representing the 600 possible quantized coordinates and tokens 601-700 representing the 100 object categories.

[0022] Optionally, as will be described in more detail, the vocabulary may also include one or more additional tokens in addition to the tokens described above.

[0023] Generally, the object detection neural network 110 is configured to generate a sequence of outputs over multiple time steps.

[0024] At each time step, neural network 110 is configured to generate a score distribution over the tokens of the vocabulary for each time step, conditional on (i) the input image and (ii) the tokens at any previous time steps in the output sequence.

[0025] Thus, at each time step during the generation of the output sequence 112, the system 100 selects a respective token at that time step in the output sequence 112 using the respective score distribution generated by the object detection neural network 110 for that time step.

[0026] As an example, the system 100 may greedily select the highest scoring token.

[0027] As another example, the system 100 may select each token by sampling the tokens according to a score distribution. As a particular example, the system may use nucleus sampling to sample the tokens according to a score distribution.

[0028] As a particular example, the object detection neural network 110 may include an encoder neural network 120 and a decoder neural network 130.

[0029] The encoder neural network 120 may be configured to process the input image 102 to generate an encoded representation 122 of the input image 102. The encoded representation 122 is a sequence including a plurality of encoded vectors that collectively represent the input image 102.

[0030] The encoder neural network 120 can be any suitable image encoder neural network that receives intensity values ​​of pixels in the image 102 and encodes those intensity values ​​into a hidden representation. Examples of such encoders include convolutional neural networks, transformer neural networks, or neural networks that include both convolutional and self-attention layers. An example of a convolutional neural network that can be used as an encoder is described in Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. An example of a Transformer Neural Network that can be used as an encoder is described in "An image is worth 16x16 words: Transformers for image recognition at scale" by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly et al., International Conference on Learning Representations, 2020.An example of a neural network containing both convolutional and self-attention layers that can be used as an encoder is described in "End-to-end object detection with transformers" by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, European Conference on Computer Vision, pages 213–229, Springer, 2020.

[0031] When the last layer of the encoder 120 is a convolutional layer that generates a feature map, the system 100 can generate the encoded representation 122 by flattening the feature map into a sequence of vectors. When the last layer of the encoder 120 is an attention layer, the system 100 can directly use the output of the attention layer as the encoded representation 122.

[0032] The decoder neural network 130 is configured to process the coded representation 122 of the input image 102 to generate the output sequence 112 .

[0033] In particular, the decoder 130 may be an autoregressive decoder neural network that, at each time step, processes tokens at any previous time steps in the output sequence 112, conditioned on the encoded representation 122 of the input image 102, to generate a score distribution for each of the time steps. The score distribution includes a respective score, e.g., a probability or logit, for each token of the vocabulary.

[0034] As a particular example, decoder 130 may be a transformer decoder that applies causal self-attention to already-generated tokens and cross-attention to encoded representations 122. That is, decoder 130 may include both a self-attention layer that applies causal self-attention to already-generated token representations and a cross-attention layer that performs cross-attention on encoded representations 122.

[0035] Examples of such transformer decoders that can be used as decoder 130 are described in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019, and Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, Improving language understanding by generative pre-training, 2018.

[0036] System 100 then generates object detection output 150 from the tokens in output sequence 112. That is, system 100 maps the tokens in output sequence 112 to data identifying one or more bounding boxes in the input image and, for each bounding box, the respective object category from a set of object categories to which the object depicted in the bounding box belongs.

[0037] For each bounding box, the data specifying the bounding box specifies the location of the bounding box within the image. As one example, the data specifying the bounding box may specify coordinates of two or more of the corners of the bounding box. As another example, the data specifying the bounding box may specify coordinates of the center of the bounding box, and the height and width of the bounding box.

[0038] Generating the object detection output 150 from the output sequence 112 is described in more detail below with reference to FIGS.

[0039] Thus, object detection neural network 110 generates an output sequence 112 of discrete tokens that can be directly mapped to an object detection output by system 100. By generating object detection output in this manner, system 100 can be easily incorporated into larger systems without the need to use neural networks that are highly customized or have complex architectures.

[0040] As a particular example, system 100 can be part of a perception system embedded within an agent that processes images and, optionally, other sensor data collected by sensors on the agent, e.g., a robot or autonomous vehicle, and object detection output can be used by the perception system or other software onboard the agent to control the agent as it navigates through an environment.

[0041] As another particular example, system 100 can be part of a perception system embedded in or in communication with different kinds of devices that process sensor data, e.g., camera surveillance systems, mobile phones, etc. The object detection output generated by system 100 can be used as part of a pre-processing stage before images are displayed to a user, or can be used to automatically trigger other actions.

[0042] As yet another particular example, a client device may interact with system 100 through an application programming interface (API), e.g., a web-based API. In particular, the client device may send an API call that includes or identifies an image to be analyzed, and system 100 may, in response, provide data that identifies an object detection output. For example, system 100 may format the object detection output in a specified format, e.g., as a JavaScript Object Notation (JSON) file or as a file in another type of data exchange format, and provide the file in response to the API call.

[0043] Before using neural network 110 to detect objects, system 100 or another training system trains neural network 110 with training data that includes multiple training examples.

[0044] Each training example includes an input training image, ground truth bounding boxes in the image, and a ground truth object detection output that identifies the respective ground truth object category for each bounding box.

[0045] Training the neural network 110 is described in more detail below with reference to FIGS.

[0046] 2 is a flow diagram of an exemplary process 200 for generating an object detection output for an input image. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an object detection system suitably programmed in accordance with this specification, such as object detection system 100 depicted in FIG. 1, may perform process 200.

[0047] The system acquires an input image (step 202).

[0048] The system processes the input images using an object detection neural network to generate an output sequence (step 204).

[0049] As described above, the output sequence includes a respective token at each of the plurality of time steps, each token selected from a vocabulary of tokens including (i) a first set of tokens, each token representing a respective discrete number from a set of discretized numbers, and (ii) a second set of tokens, each token representing a respective object category from a set of object categories.

[0050] The vocabulary may also optionally include additional tokens.

[0051] For example, a vocabulary may include an end-of-sequence (EOS) token.

[0052] As another example, the vocabulary may include a "noise" token, which represents a noise object category that is not included in the set of object categories (and therefore does not represent any actual object that may be depicted in the input image). The "noise" token may be added to the vocabulary before training the neural network to improve the effectiveness of the training, as described in more detail below.

[0053] Generally, an object detection neural network is configured to generate a sequence of outputs over multiple time steps.

[0054] At each time step, the neural network is configured to generate a score distribution over the tokens of the vocabulary for each time step, conditional on (i) the input image and (ii) the tokens at any previous time steps in the output sequence.

[0055] Thus, at each time step during the generation of the output sequence, the system selects a respective token at that time step in the output sequence using the respective score distribution generated by the object detection neural network for that time step.

[0056] As an example, the system may greedily select the highest scoring token.

[0057] As another example, the system may select each token by sampling the tokens according to a score distribution. As a particular example, the system may use kernel sampling to sample the tokens according to a score distribution.

[0058] As yet another example, when the vocabulary includes a noise token, the system may determine at any given time step whether the noise token is selected, and if so, replace the noise token with the highest-scoring token from the second set of tokens. That is, in response to determining that a noise token is selected, the system selects, from only the tokens in the second set of tokens, the token with the highest score to replace the noise token.

[0059] In some implementations, the system continues to add tokens to the output sequence until an end-of-sequence (EOS) token is selected. In some other implementations, the system continues to add tokens to the output sequence until the output sequence has a fixed length, i.e., until it has a maximum number of tokens. In still other implementations, the system continues to add tokens to the output sequence until an EOS token is selected or until the output sequence has a fixed length, whichever occurs first.

[0060] From the tokens in the output sequence, the system generates object detection outputs, i.e., data identifying one or more bounding boxes in the input image and, for each bounding box, the respective object category from the set of object categories to which the object depicted in the bounding box belongs (step 206).

[0061] In particular, due to the way the neural network is trained, the output sequence contains a respective subsequence for each of the one or more bounding boxes in the input image.

[0062] A subsequence for a given bounding box includes a token from a first set of tokens and a token from a second set of tokens. For example, the subsequence may include a total of five tokens: four tokens from the first set and one token from the second set. As a particular example, the subsequence may include four tokens from the first set followed by one token from the second set.

[0063] In some cases, the four discrete numbers represented by the four tokens from the first set specify the coordinates in the input image of two corners of the bounding box, e.g., the (x, y) coordinates of the top-left and bottom-right corners, or the bottom-right and top-left corners.

[0064] In some other cases, the four discrete numbers represented by the four tokens from the first set specify the coordinates in the input image of the center of the bounding box, and the height and width of the bounding box.

[0065] Thus, to generate the object detection output, for each subsequence in the output sequence, the system can identify the coordinates of a bounding box in the input image from the tokens in the subsequence that belong to the first set of tokens, and then identify the object category represented by the tokens in the corresponding subsequence that belong to the second set of tokens as the respective object category to which the object depicted in the bounding box belongs.

[0066] In other words, for each first token in the subsequence, the system may map the first token to a quantized coordinate value represented by the token to generate coordinates that define a bounding box, and for each second token, the system may map the token to a label or other data that identifies the object category represented by the second token.

[0067] In some implementations, the system further associates with the bounding box a respective score assigned to a token representing each object category of the bounding box of the score distribution at the corresponding time step to indicate the confidence that the respective object category is the correct category of the object.

[0068] In some implementations, the system may refrain from including one or more bounding boxes in the object detection output if, for example, the confidence score of the corresponding object category is below a threshold.

[0069] The system can then output an object detection output, i.e., data identifying a bounding box, an object category, and optionally, an associated confidence score.

[0070] As a particular example, when the system is part of a perception system embedded within an agent that processes images and, optionally, other sensor data collected by sensors in the agent, e.g., a robot or autonomous vehicle, the system can provide data identifying object detection outputs to the perception system or other software onboard the agent in order to control the agent as it navigates through its environment.

[0071] As another particular example, when the system is part of a perception system that is integrated into or communicates with different types of devices that process sensor data, e.g., camera surveillance systems, mobile phones, etc., the system may output data to another software component of the device for use in pre-processing images before they are displayed to a user, or for use in automatically triggering an action, e.g., an alert.

[0072] As yet another particular example, when a client device can interact with the system through an application programming interface (API), e.g., a web-based API, the system can provide data identifying the object detection output in response to the API call. For example, the system can format the object detection output in a specified format, e.g., as a JavaScript Object Notation (JSON) file or as a file in another type of data exchange format, and provide the file in response to the API call.

[0073] FIG. 3 shows an example object detection output extracted from the output sequence produced by an object detection neural network.

[0074] In particular, Figure 3 shows portions of example object detection output for three example input images 310, 320, and 330. Each portion contains information specifying one of the bounding boxes in the image.

[0075] As can be seen in FIG. 3, the system processes each of the input images 310, 320, and 330 using the neural network 110 to generate a respective output sequence, and then extracts corresponding object detection output portions 312, 314, and 316 from the output sequence for the input image.

[0076] For example, object detection output portion 312 may specify that the bounding box has a lower right corner at (in pixel coordinates) y_min=9 and x_min=7 and an upper right corner at y_max=67 and x_max=98, indicating an image of an object belonging to the "train" category. For example, if tokens are represented as integers, the image is 100x100, and each first token corresponds to one pixel in the image, the system may extract the object detection output portion from the subsequence [9, 7, 67, 98, 115], where "115" is the token representing the "train" category. That is, although portion 312 is shown as including identification information for each element, the underlying output sequence is simply a sequence of discrete tokens from a vocabulary.

[0077] 4 is a flow diagram of an example process 400 for training an object detection neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a training system suitably programmed in accordance with this specification, such as object detection system 100 depicted in FIG. 1, or a different system of one or more computers at one or more locations, may perform process 400.

[0078] The system can repeatedly perform iterations of process 400 with different batches of training examples to train the neural network, i.e., to iteratively adjust the values ​​of the neural network's parameters. That is, in each iteration of process 400, the system obtains one or more batches of training examples, e.g., by sampling the batches from a larger set of training data, and then performs an iteration of process 400 to update the current values ​​of the network parameters at the time of the iteration.

[0079] For example, the system may continue to perform iterations of process 400 until a termination criterion is met, e.g., until a threshold number of training iterations have been performed, until a specified amount of time has elapsed, or until the parameters are determined to converge.

[0080] The system obtains a batch of training examples (step 402). Each training example includes a training image and one or more ground truth bounding boxes in the training image and a target output that identifies a ground truth object category for each bounding box. The ground truth object category is the object category from the set of object categories to which the object depicted in the ground truth bounding box was classified.

[0081] In some implementations, the system applies one or more augmentation techniques to generate batches from an initial batch of training examples.

[0082] As an example, the system can generate one or more of the batches of training images by applying one or more image augmentation policies to corresponding initial training images. The system can then associate each generated training image with the target output of the corresponding initial training image. Applying the image augmentation policies can increase the robustness of the trained neural network to various image perturbations that may not be well represented in the training data.

[0083] For example, an image augmentation policy may specify how random scaling, cropping, or other image augmentation techniques are applied to initial training images to generate batches of training examples.

[0084] As an example, the system may perform scale jittering with random cropping on the initial training images. An example of such a technique is described in "Simple copy-paste is a strong data augmentation method for instance segmentation" by Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 2918-2928, 2021.

[0085] As another example, the system may perform image scaling by resizing the image so that its long side has a fixed number of pixels (with a fixed aspect ratio).

[0086] As another example, the system can apply color distortion to the intensity values ​​of pixels in the initial training images. An example of such a technique is described in Andrew G Howard, Some improvements on deep convolutional neural network based image classification, arXiv preprint arXiv:1312.5402, 2013.

[0087] Instead of, or in addition to, image robustness, the system can also apply augmentation to the target output of training examples to increase the robustness of the trained model to noisy predictions or mislabeled training data.

[0088] For example, the system may select one or more of the initial ground truth bounding boxes in a particular initial training image of a particular one of the initial training examples, and then generate a corresponding ground truth bounding box for the corresponding training example by applying noise to the initial ground truth bounding box in the particular training image for each selected initial ground truth bounding box. For example, the system may apply random scaling to the initial ground truth bounding box, may apply random shifts to the initial ground truth bounding box, or may apply both.

[0089] For each training image, the system generates a target output sequence (step 404) that includes, for each ground truth bounding box, a respective subsequence that includes (i) a first set of tokens that define the location of the bounding box in the image and (ii) a second token that represents the ground truth object category of the bounding box. That is, the target output sequence is composed of one or more subsequences, each corresponding to a different ground truth bounding box. Optionally, the last subsequence in the target output sequence can be followed by an end-of-sequence token, which is the last token in the target output sequence.

[0090] During inference, the order in which bounding boxes are arranged in the output sequence produced by the neural network does not matter, i.e., an output that identifies bounding box A before bounding box B in the sequence should be considered the same as an output that places bounding box B earlier in the sequence, and the system can generate a target output sequence that reflects this. In particular, the system can order each subsequence in a random order within the target output sequence.

[0091] In some implementations, the system may also make one or more modifications to the target output sequence for any given training image to improve the performance of the neural network after training. As described above, when a neural network is trained using "teacher forcing," the system also modifies the corresponding input sequence provided to the decoder neural network during training. Modifying the target output sequence (and optionally the corresponding input sequence) is referred to as "sequence augmentation."

[0092] For example, as described above, in some implementations, the token vocabulary may include tokens representing "noise" object categories that are not in the set of object categories. In these examples, the system may generate one or more random bounding boxes in the training images and, for each random bounding box, include in the target output sequence a corresponding "noise" subsequence that includes (i) a first set of tokens that define the location of the random bounding box in the training images and (ii) a second token that represents the noise object category that is not in the set of object categories.

[0093] As a particular example, the system may add a noise subsequence after the last subsequence that corresponds to one of the ground truth bounding boxes.

[0094] In some implementations, the system can add a fixed number of noise subsequences to each target output sequence. In some other implementations, the system can add noise subsequences so that each target output sequence contains the same fixed number of subsequences (i.e., the same number of ground truth plus noise subsequences).

[0095] Adding these random (or "noisy") bounding boxes to the target sequence can improve the performance of the trained neural network in various ways. For example, some of these noise objects may be identical to or overlap with some of the ground truth objects, simulating noisy and overlapping predictions, i.e., thereby training the neural network to be more robust against these types of predictions in the training data. As another example, introducing noisy bounding boxes into the target output sequence can prevent the neural network from terminating without identifying all objects in the image during inference, without also introducing noisy and overlapping predictions into the output generated as inference (as can occur when some other techniques, such as artificially reducing the scores assigned to EOS tokens, are used).

[0096] The system then trains an object detection neural network to maximize, for each training image and for each token of at least a subset of the tokens in the target output sequence for the training image, the log-likelihood of the token conditional on any preceding tokens in the target output sequence and the training image (step 406).

[0097] When the target output sequence does not include a noisy bounding box, the "at least a subset" may include all of the tokens in the target output sequence. When the target output sequence does include a noisy bounding box, the neural network is not trained to maximize the log-likelihood of the tokens in the first set of tokens relative to a random bounding box (but is trained to maximize the log-likelihood of the second token representing the noise category). That is, the "at least a subset" includes all of the tokens except the first set of tokens relative to the random bounding box. This allows the neural network to learn to identify the noisy bounding box rather than imitate it.

[0098] To train the neural network to maximize the log-likelihood, the system uses backpropagation to calculate the gradient of an objective function that measures the log-likelihood of at least a subset of tokens with respect to the parameters of the encoder neural network and the decoder neural network, and then updates the parameters using the determined gradient. For example, the system can apply an appropriate optimizer, such as the Adam optimizer, the rmsProp optimizer, the Adafactor optimizer, or a different machine learning optimizer, to the gradient and parameters to update the parameters. For example, the loss function can be a combination, such as a sum or weighted average, of the log-likelihoods of the training output sequences of the training examples for each training example. When the target output sequence does contain a noisy bounding box, the first set of tokens associated with the random bounding box is not included in the combination.

[0099] FIG. 5 shows an example of training a neural network for a given target output sequence.

[0100] In particular, Figure 5 shows a first example 510 of training a neural network without sequence augmentation and a second example 512 of training a neural network with sequence augmentation. More specifically, the first example 510 shows a target output sequence 520 and a corresponding input sequence 530. During training, the training system uses "teacher forcing," such that the neural network processes the input sequence 530 to generate a respective score distribution for each position in the target output sequence 520 that is used to train the neural network, as described above. Due to the "causality" employed by the decoder, the corresponding input sequence 530 is shifted by one token relative to the target output sequence 520, and therefore, each score distribution for each given position in the target output sequence 520 depends on the token at the position preceding the given position in the target output sequence 520 (and always on the initial "start" token provided as the first input at the first time step during autoregressive generation at inference). This dependency is shown by arrows in examples 510 and 512, where each token in the target output sequence depends on each token in the corresponding input sequence that is connected to it by an arrow, i.e., the corresponding input sequence processed by the autoregressive decoder at each generation time step.

[0101] In particular, in the first example 510, the target output sequence 520 includes ten tokens y1 through y10 followed by an EOS token ("end") and no additional tokens.

[0102] The second example 512 also includes a target output sequence 540 and a corresponding input sequence 550. However, in the target output sequence 540, token y10 is followed by tokens for two “noisy” bounding boxes. In particular, the target output sequence 540 includes two first sets of tokens representing the coordinates of the noise bounding boxes, each of which is followed by a “noise” token indicating that the bounding box is a noise bounding box that was not originally present in the corresponding input image. Similarly, the corresponding input sequence 550 also includes an additional token following token y10. The first set of tokens for the noisy bounding boxes are labeled “n / a” in FIG. 5 because their log-likelihoods are not considered when training the neural network, e.g., their loss is set to zero.

[0103] This specification uses the term "configured" in reference to systems and computer program components. One or more computer systems configured to perform particular operations or actions means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation. One or more computer programs configured to perform particular operations or actions means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0104] Embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiver suitable for execution by a data processing apparatus.

[0105] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include dedicated logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Optionally, an apparatus can include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0106] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple organized files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and connected together by a data communications network.

[0107] As used herein, the term "database" is used broadly to refer to any collection of data; that is, the data need not be structured in any particular way, or at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0108] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and running on the same computer or computers.

[0109] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0110] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from, transfer data to, or both of, the mass storage devices. However, a computer need not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0111] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0112] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone, running a messaging application, and receiving a reply message from the user in return.

[0113] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling the often computationally intensive parts of machine learning training or generation, e.g., inference, workloads.

[0114] The machine learning model may be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0115] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0116] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device for the purpose of, e.g., displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0117] While this specification contains many specific implementation details, these should not be considered limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of a subcombination.

[0118] Similarly, while operations are shown in the figures and recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the operations shown be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems generally can be integrated together in a single software product or packaged in multiple software products.

[0119] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0120] 100 Object Detection System 102 input images 110 Object Detection Neural Network 112 Output Sequence 120 Encoder Neural Network 122 Coded Representation 130 Decoder Neural Network 150 object detection outputs 200 processes 310 input images 312 Object detection output part 314 Object detection output part 316 Object detection output part 320 input images 330 input images 400 processes 510 First Example 512 Second Example 520 Target Output Sequence 530 Input Sequence 540 Target Output Sequence 550 corresponding input sequences

Claims

1. A method implemented by one or more computers, comprising: obtaining an input image; processing the input image using an object detection neural network to generate an output sequence comprising a respective token at each of a plurality of time steps; The processing step includes: processing the input image using an encoder neural network to generate an encoded representation of the input image; processing the coded representation of the input image using a decoder neural network to generate the output sequence; each token is selected from a vocabulary of tokens comprising: (i) a first set of tokens, each token representing a respective discrete number from a set of discretized numbers; and (ii) a second set of tokens, each token representing a respective object category from a set of object categories; generating from the tokens in the output sequence one or more bounding boxes in the input image and, for each bounding box, data identifying a respective object category from the set of object categories to which an object depicted in the bounding box belongs, wherein a token in the output sequence selected from the first set of tokens defines a position of the bounding box in the input image.

2. the output sequence includes a respective sub-sequence corresponding to each of the one or more bounding boxes, and the generating the data identifying the one or more bounding boxes includes, for each bounding box: determining coordinates of the bounding box in the input image from tokens in the corresponding subsequence belonging to the first set of tokens; and identifying the object categories represented by tokens in the corresponding subsequence of tokens belonging to the second set of tokens as the respective object categories to which the object depicted within the bounding box belongs.

3. 3. The method of claim 2, wherein each subsequence includes four tokens from the first set of tokens, and four discrete numbers represented by the four tokens specify coordinates in the input image of two corners of the bounding box.

4. 3. The method of claim 2, wherein each subsequence includes four tokens from the first set of tokens, and the four discrete numbers represented by the four tokens specify the coordinates in the input image of the center of the bounding box and the height and width of the bounding box.

5. The object detection neural network 10. The method of claim 1, wherein the method is trained to maximize the log-likelihood of a token conditional on a preceding token in an output sequence generated from a training image and the training image.

6. the object detection neural network is configured to generate respective score distributions over the tokens of the vocabulary at each time step conditional on (i) the input image and (ii) the tokens at any previous time steps in the output sequence, and processing the input images using the object detection neural network to generate an output sequence includes, for each time step:

2. The method of claim 1, comprising selecting the respective tokens at the time step in the output sequence using the respective score distributions generated by the object detection neural network for the time step.

7. The method of claim 6 , wherein selecting the respective tokens comprises selecting the tokens with the highest scores in the respective score distributions.

8. The method of claim 6 , wherein selecting each token comprises sampling tokens according to the score distribution.

9. The method of claim 8 , wherein selecting each token comprises sampling tokens according to the score distribution using kernel sampling.

10. the vocabulary includes noise tokens representing noise categories not included in the set of object categories, and processing the input images using the object detection neural network to generate an output sequence comprises, for a particular one of the time steps: determining that the token with the highest score for the particular time step is the noise token; and in response thereto, selecting the token having the highest score from only the tokens in the second set of tokens.

11. The decoder neural network calculates, for each time step, 7. The method of claim 6, configured to process the tokens at any previous time steps in the output sequence while conditioned on the encoded representation of the input image to generate a score distribution for each of the time steps.

12. 7. The method of claim 6, further comprising associating, for each of the one or more bounding boxes, a respective score assigned to the token representing the respective object category of the bounding box in the score distribution at a corresponding time step to represent a confidence that the respective object category is a correct category for the object.

13. 2. The method of claim 1, further comprising outputting the data identifying the one or more bounding boxes in the input image and, for each bounding box, the respective object category from the set of object categories to which the object depicted in the bounding box belongs.

14. 10. A method for training an object detection neural network according to claim 1, comprising: obtaining a batch of training images and, for each training image, a target output that identifies one or more ground truth bounding boxes in the image and a respective ground truth object category for each bounding box; for each training image, generating a target output sequence including, for each ground truth bounding box, a respective sub-sequence including (i) a first set of tokens defining the location of the bounding box within the image, and (ii) a second set of tokens representing the ground truth object category of the bounding box; and training the object detection neural network to maximize, for each training image, and for each token of at least a subset of tokens in the target output sequence for the training image, the log-likelihood of the token conditional on any preceding tokens in the target output sequence and the training image.

15. obtaining a batch of training images and, for each training image, a target output that identifies one or more ground truth bounding boxes in the image and a respective ground truth object category for each bounding box, comprising:

15. The method of claim 14, comprising generating one or more of the training images of the batch by applying one or more image augmentation policies to corresponding initial training images.

16. obtaining a batch of training images and, for each training image, a target output that identifies one or more ground truth bounding boxes in the image and a respective ground truth object category for each bounding box, comprising:

15. The method of claim 14, comprising, for a particular bounding box in a particular training image, generating the bounding box by applying noise to an initial ground truth bounding box in the particular training image.

17. generating a target output sequence for each training image, generating one or more random bounding boxes within the training images; 15. The method of claim 14, comprising including in the target output sequence, for each random bounding box, (i) a first set of tokens that define a location of the random bounding box within the training images, and (ii) a second token that represents a noise object category that is not in the set of object categories.

18. 18. The method of claim 17, wherein the object detection neural network is not trained to maximize the log-likelihood of tokens in the first set of tokens with respect to the random bounding box.

19. generating a target output sequence for each training image, 15. The method of claim 14, comprising ordering the respective subsequences in a random order within the target output sequence.

20. one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 19.

21. 20. A computer program comprising instructions that, when executed by one or more computers, cause said one or more computers to perform each of the operations of the method of any one of claims 1 to 19.

Images