Object detection in an image by generating a sequence of tokens.

Abstract

Objects in an input image are detected by processing the image using a neural network to generate a token output sequence. [Solution] In one embodiment, one of the methods is a step of acquiring an input image, processing the input image using an object detection neural network to generate an output sequence containing each token in each of a plurality of time steps, wherein each token is selected from a vocabulary of tokens comprising (i) a first set of tokens each representing a discrete number from a set of discretized numbers, and (ii) a second set of tokens each representing an object category from a set of object categories, and generating an object detection output relating to the input image from the tokens in the output sequence.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 245,783, filed on September 17, 2021. The disclosure of the prior application is considered to be a part of the disclosure of this application and is incorporated herein by reference.

[0002] This specification relates to processing an input using a neural network.

Background Art

[0003] A neural network is a machine - learning model that uses one or more layers of non - linear units to predict an output regarding an input received. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer within the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from the input received according to the current values of each set of parameters.

Prior Art Documents

Non - Patent Documents

[0004]

Non - Patent Document 1

Non - Patent Document 2

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that detects objects in an input image by processing the image using a neural network to generate a token output sequence.

[0006] The subjects described herein may be implemented in specific embodiments to achieve one or more of the following advantages:

[0007] Existing object detection methods require a high degree of customization, possess complex architectures, and are difficult to integrate into larger systems. The technique described, on the other hand, uses an object detection neural network that generates a sequence of discrete tokens corresponding to the description of an object. The object detection neural network has a simple architecture and is therefore easily integrated into perceptual systems and easily extensible to different domains or applications. Furthermore, despite its simple architecture, because the technique described generates a sequence of discrete tokens corresponding to the description of an object (bounding box and class label), the object detection neural network can achieve performance comparable to or exceeding that of much more complex systems highly designed for specific object detection tasks.

[0008] Details of one or more embodiments covered by this specification are described in the accompanying drawings and the description below.

[0009] Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating an example object detection system. [Figure 2] This is a flowchart illustrating an exemplary process for detecting objects in an input image. [Figure 3] This figure shows an example of object detection output generated using an object detection neural network. [Figure 4] This is an illustrative process flowchart for training an object detection neural network. [Figure 5] This figure shows the training of an object detection neural network in a training example. [Modes for carrying out the invention]

[0011] Like reference numerals and reference designations in the various drawings indicate like elements.

[0012] FIG. 1 is a diagram of an exemplary object detection system 100. The object detection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described below may be implemented.

[0013] The object detection system 100 is a system that receives an input image 102 and detects objects in the input image 102 by processing the image 102 using an object detection neural network 110, thereby generating an object detection output 150 for the input image 102.

[0014] The object detection output 150 identifies one or more bounding boxes in the input image 102 that each correspond to a detected object, i.e., that are predicted to contain a depiction of the detected object, and for each of the bounding boxes, identifies an object category from a set of object categories to which the detected object within the bounding box belongs.

[0015] More specifically, the system 100 acquires the input image 102.

[0016] The system 100 processes the input image 102 using the object detection neural network 110 to generate an output sequence 112 that includes a plurality of tokens, i.e., processes the intensity values of the pixels of the input image 102.

[0017] Each token within the sequence is selected from a vocabulary of tokens that includes (i) a first set of tokens each representing a discrete number from a set of discretized numbers, and (ii) a second set of tokens each representing a respective object category from a set of object categories. The tokens of the vocabulary can be represented in any suitable way, for example, as distinguishable integers or other alphanumeric symbols from each other.

[0018] More specifically, the system discretizes the numbers used to represent the coordinates of any given point within the input image 102 into a plurality of bins, and each bin corresponds to a respective token of the first set of tokens. As a specific example, the system can discretize the range of values between zero and the height or width of the image in pixel units into a determined number of equally spaced bins, such that as a result, each bin corresponds to a different subset of pixel indices that can be used to represent the coordinates of points (in pixel units) within the image. Thus, if the input image is 600 pixels × 600 pixels and there are 600 bins, each bin corresponds to a different pixel index from 1 to 600. If the input image is 600×600 and there are 300 bins, each bin corresponds to a different set of two pixel indices from 1 to 600.

[0019] Thus, each token of the first set of tokens represents a different bin in the discretization of the possible coordinate values of pixels within the image and can be mapped to different quantized coordinate values, for example, the representative value of the bin represented by the token. For example, the representative value can be the average of the endpoints of the bin or one of the two endpoints of the bin. This quantization method of coordinates enables the system to use a relatively small vocabulary to represent possible pixel coordinates while maintaining high accuracy.

[0020] Regarding the second set of tokens, the system can assign a different unique token to each object category of the set.

[0021] Therefore, as a specific example, if there are 600 bins in a quantization scheme and 100 object categories, and the tokens are represented as integers, the vocabulary may include tokens 1-600 representing 600 possible quantized coordinates and tokens 601-700 representing 100 object categories.

[0022] Optionally, as will be explained in more detail, the vocabulary may include one or more additional tokens in addition to the tokens described above.

[0023] Generally, the object detection neural network 110 is configured to generate an output sequence over multiple time steps.

[0024] At each time step, the neural network 110 is configured to generate a score distribution on tokens in the vocabulary for each time step, conditional on (i) the input image and (ii) the tokens in any previous time step in the output sequence.

[0025] Therefore, at each time step during the generation of the output sequence 112, the system 100 selects each token in the output sequence 112 for that time step using the respective score distribution generated by the object detection neural network 110 for that time step.

[0026] For example, system 100 may greedily select the token with the highest score.

[0027] As another example, system 100 may select each token by sampling the tokens according to a score distribution. In a specific example, the system may use nucleus sampling to sample tokens according to a score distribution.

[0028] As a specific example, the object detection neural network 110 may include an encoder neural network 120 and a decoder neural network 130.

[0029] The encoder neural network 120 may be configured to process the input image 102 to generate an encoded representation 122 of the input image 102. The encoded representation 122 is a sequence containing a plurality of encoded vectors that collectively represent the input image 102.

[0030] The encoder neural network 120 can be any suitable image encoder neural network that receives the intensity values ​​of pixels in image 102 and encodes those intensity values ​​into a hidden representation. Examples of such encoders include convolutional neural networks, transformer neural networks, or neural networks that include both convolutional and self-attention layers. An example of a convolutional neural network that can be used as an encoder is described in Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. An example of a transformer neural network that can be used as an encoder is described in "An image is worth 16x16 words: Transformers for image recognition at scale" by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly, presented at the International Conference on Learning Representations, 2020.An example of a neural network that includes both convolutional and self-attention layers and can be used as an encoder is described in *European Conference on Computer Vision*, pp. 213–229, Springer, 2020, by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, *End-to-end object detection with transformers*.

[0031] When the last layer of encoder 120 is a convolutional layer that generates a feature map, system 100 can generate an encoded representation 122 by flattening the feature map into a sequence of vectors. When the last layer of encoder 120 is an attention layer, system 100 can directly use the output of the attention layer as the encoded representation 122.

[0032] The decoder neural network 130 is configured to process the encoded representation 122 of the input image 102 in order to generate the output sequence 112.

[0033] In particular, the decoder 130 can be an autoregressive decoder neural network that processes tokens in any previous time step in the output sequence 112, while in each time step, it is conditional on the encoded representation 122 of the input image 102 to generate the respective score distribution for the time step. The score distribution includes the respective score for each token in the vocabulary, for example, a probability or logit.

[0034] As a specific example, the decoder 130 can be a transformer decoder that applies causal self-attention to already generated tokens and cross-attention to the encoded representation 122. That is, the decoder 130 may include both a self-attention layer that applies causal self-attention to the representation of already generated tokens and a cross-attention layer that performs cross-attention on the encoded representation 122.

[0035] Examples of such transformer decoders that can be used as decoder 130 are described in Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019, as well as in Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, Improving language understanding by generative pre-training, 2018.

[0036] At that time, system 100 generates an object detection output 150 from the tokens in output sequence 112. That is, system 100 maps the tokens in output sequence 112 to data that identifies one or more bounding boxes in the input image and, for each bounding box, each object category from a set of object categories to which the object drawn within the bounding box belongs.

[0037] For each bounding box, the data identifying the bounding box specifies its position within the image. For example, the data identifying the bounding box may specify the coordinates of two or more of the bounding box's corners. Alternatively, the data identifying the bounding box may specify the coordinates of the bounding box's center, as well as its height and width.

[0038] The generation of the object detection output 150 from the output sequence 112 is described in more detail below with reference to Figures 2 and 3.

[0039] Therefore, the object detection neural network 110 generates an output sequence 112 of discrete tokens that can be directly mapped to the object detection output by system 100. By generating the object detection output in this way, system 100 can be easily integrated into larger systems without requiring the use of a highly customized or complex neural network architecture.

[0040] As a specific example, system 100 could be part of a perception system built into an agent that processes images and optionally other sensor data collected by sensors of an agent, such as a robot or autonomous vehicle, and the object detection output could be used by the perception system or other software installed in the agent to control the agent as it navigates through the environment.

[0041] As another specific example, system 100 could be part of a perception system that is embedded in or communicates with different types of devices that process sensor data, such as camera surveillance systems or mobile phones. The object detection output generated by system 100 can be used as part of a pre-processing stage before the image is displayed to the user, or it can be used to automatically trigger other actions.

[0042] As yet another specific example, a client device may interact with system 100 through an application programming interface (API), such as a web-based API. In particular, a client device may send an API call that includes or identifies an image to be analyzed, and system 100 may respond by providing data that identifies the object detection output. For example, system 100 may format the object detection output in a specified format, such as a JavaScript Object Notation (JSON) file, or a file in another type of data exchange format, and provide the file in response to the API call.

[0043] Before using the neural network 110 to detect objects, system 100 or another training system trains the neural network 110 with training data that includes multiple training examples.

[0044] Each training example includes an input training image and a ground truth object detection output that identifies ground truth bounding boxes within the image and the respective ground truth object category for each bounding box.

[0045] Training the neural network 110 is described in more detail below with reference to Figures 4 and 5.

[0046] Figure 2 is a flowchart of an exemplary process 200 for generating object detection output for an input image. For convenience, the process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, an object detection system appropriately programmed according to this specification, such as the object detection system 100 depicted in Figure 1, may perform the process 200.

[0047] The system acquires the input image (step 202).

[0048] The system processes the input image using an object detection neural network to generate an output sequence (step 204).

[0049] As described above, the output sequence contains each token at each of the multiple time steps. Each token is selected from a vocabulary of tokens that includes (i) a first set of tokens, each representing a discrete number from a set of discrete numbers, and (ii) a second set of tokens, each representing an object category from a set of object categories.

[0050] The vocabulary is optional and may include additional tokens.

[0051] For example, a vocabulary may include end-of-sequence (EOS) tokens.

[0052] As another example, a vocabulary can include "noise" tokens, which represent noise object categories that are not included in the set of object categories (and therefore do not represent any actual objects that may be depicted in the input image). Noise tokens can be added to the vocabulary before training the neural network to enhance the training effect, as will be explained in more detail below.

[0053] Generally, object detection neural networks are configured to generate output sequences over multiple time steps.

[0054] At each time step, the neural network is configured to generate a score distribution on tokens in the vocabulary for that time step, conditional on (i) the input image and (ii) the tokens in any previous time step within the output sequence.

[0055] Therefore, at each time step during the generation of the output sequence, the system selects each token in the output sequence for that time step using the respective score distribution generated by the object detection neural network for that time step.

[0056] For example, the system could greedily select the token with the highest score.

[0057] As another example, the system could select each token by sampling them according to a score distribution. In a specific example, the system could use nuclear sampling to sample tokens according to a score distribution.

[0058] As yet another example, when the vocabulary contains noise tokens, the system determines whether the noise token was selected at any given time step, and if so, may replace the noise token with the highest-scoring token from a second set of tokens. That is, in response to the determination that a noise token was selected, the system selects the highest-scoring token to replace the noise token from only the tokens in the second set of tokens.

[0059] In some implementations, the system continues to add tokens to the output sequence until an End of Sequence (EOS) token is selected. In some other implementations, the system continues to add tokens to the output sequence until the output sequence reaches a fixed length, i.e., until it has the maximum number of tokens. In yet another implementation, the system continues to add tokens to the output sequence until an EOS token is selected or until the output sequence reaches a fixed length, regardless of which happens first.

[0060] The system generates object detection output from the tokens in the output sequence, i.e., data that identifies one or more bounding boxes in the input image and, for each bounding box, the respective object category from a set of object categories to which the object depicted within the bounding box belongs (step 206).

[0061] In particular, thanks to the way the neural network is trained, the output sequence contains a separate subsequence for each of the one or more bounding boxes in the input image.

[0062] A subsequence with respect to a given bounding box includes tokens from a first set of tokens and tokens from a second set of tokens. For example, a subsequence may include a total of five tokens, i.e., four tokens from the first set and one token from the second set. In a specific example, a subsequence may include four tokens from the first set followed by one token from the second set.

[0063] In some cases, four discrete numbers represented by four tokens from the first set specify the (x, y) coordinates of two corners of the bounding box in the input image, for example, the top-left and bottom-right corners, or the bottom-right and top-left corners.

[0064] In some other cases, four discrete numbers represented by four tokens from the first set specify the coordinates of the center of the bounding box in the input image, as well as the height and width of the bounding box.

[0065] Therefore, in order to generate object detection output, the system can, for each subsequence in the output sequence, identify the coordinates of the bounding box in the input image from the tokens in the subsequence belonging to the first set of tokens, and then identify the object category represented by the tokens in the corresponding subsequence belonging to the second set of tokens, as the respective object category to which the object drawn within the bounding box belongs.

[0066] In other words, for each first token in a subsequence, the system can map the first token to the quantized coordinate value represented by the token to generate coordinates that define a bounding box, and for each second token, the system can map the token to a label or other data that identifies the object category represented by the second token.

[0067] In some implementations, the system further associates the respective scores assigned to tokens representing each object category in the bounding box of the score distribution at the corresponding time step with the bounding box, in order to represent the confidence that each object category is the correct category of the object.

[0068] In some implementations, the system may, for example, refrain from including one or more bounding boxes in the object detection output if the confidence score of the corresponding object category is below a threshold.

[0069] The system can then output object detection output, which includes data identifying the bounding box, the object category, and optionally, an associated confidence score.

[0070] As a specific example, when the system is part of a perception system embedded within an agent that processes images and, optionally, other sensor data collected by an agent, such as a robot or autonomous vehicle's sensors, the system can provide data to the perception system or other software installed in the agent to identify object detection outputs in order to control the agent as it navigates through the environment.

[0071] As another specific example, when a system is part of a perception system that is embedded in or communicates with different types of devices that process sensor data, such as camera surveillance systems or mobile phones, the system may output data to other software components of the devices for use in pre-processing images before they are displayed to a user, or for use in automatically triggering actions, such as warnings.

[0072] As yet another specific example, when a client device can interact with the system through an application programming interface (API), such as a web-based API, the system can provide data identifying object detection output in response to an API call. For example, the system can format the object detection output in a specified format, such as a JavaScript Object Notation (JSON) file, or a file in another type of data exchange format, and provide the file in response to an API call.

[0073] Figure 3 shows an exemplary object detection output extracted from the output sequence generated by the object detection neural network.

[0074] In particular, Figure 3 shows portions of exemplary object detection output for three exemplary input images 310, 320, and 330. Each portion contains information that specifies one of the bounding boxes in the image.

[0075] As can be seen in Figure 3, the system processes each of the input images 310, 320, and 330 using the neural network 110 to generate their respective output sequences, and then extracts the corresponding object detection output portions 312, 314, and 316 from the output sequences for the input images.

[0076] For example, the object detection output portion 312 explicitly indicates that the bounding box has its lower right corner at y_min=9 and x_min=7 (in pixel coordinates) and its upper right corner at y_max=67 and x_max=98, and that it is an image of an object belonging to the "train" category. For example, if tokens are represented as integers, the image is 100x100, and each first token corresponds to one pixel of the image, the system can extract the object detection output portion from the subsequence [9, 7, 67, 98, 115], where "115" is the token representing the "train" category. That is, portion 312 is shown as containing identification information for each element, but the underlying output sequence is merely a sequence of discrete tokens from the vocabulary.

[0077] Figure 4 is a flowchart of an exemplary process 400 for training an object detection neural network. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a training system appropriately programmed according to this specification, such as the object detection system 100 depicted in Figure 1, or different systems of one or more computers in one or more locations may perform process 400.

[0078] The system can repeatedly run iterations of process 400 with different batches of training examples to train a neural network, that is, to iteratively adjust the values ​​of the neural network's parameters. In each iteration of process 400, the system obtains one or more batches of training examples, for example, by sampling a batch from a larger set of training data, and then runs iterations of process 400 to update the current values ​​of the network parameters at the time of the iteration.

[0079] For example, the system may continue iterating through process 400 until a termination criterion is met, such as until a certain number of training iterations have been performed, until a specified amount of time has elapsed, or until it is determined that the parameters have converged.

[0080] The system acquires a batch of training examples (step 402). Each training example includes a training image and a target output that identifies one or more ground truth bounding boxes within the training image and the respective ground truth object category for each bounding box. A ground truth object category is an object category from a set of object categories to which objects drawn within a ground truth bounding box are classified.

[0081] In some implementations, the system applies one or more augmentation techniques to generate batches from an initial batch of training examples.

[0082] For example, the system can generate one or more training images from a batch by applying one or more image augmentation policies to the corresponding initial training images. The system can then associate each generated training image with the target output of the corresponding initial training image. Applying image augmentation policies can improve the robustness of the trained neural network against various image perturbations that may not be well represented in the training data.

[0083] For example, an image augmentation policy can specify how random scaling, cropping, or other image augmentation techniques should be applied to the initial training images to generate batches of training examples.

[0084] For example, the system may perform scale jittering with random cropping on the initial training images. An example of such a technique is described in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 2918–2928, 2021, by Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph, "Simple copy-paste is a strong data augmentation method for instance segmentation."

[0085] As another example, a system can perform image scaling by resizing the image so that its longer side has a fixed number of pixels (at a fixed aspect ratio).

[0086] As another example, the system can apply color distortion to the pixel intensity values ​​of the initial training images. An example of such a technique is described in Andrew G Howard, Some improvements on deep convolutional neural network based image classification, arXiv preprint arXiv:1312.5402, 2013.

[0087] Instead of image rigidity, or in addition to image augmentation, the system can also apply augmentation to the target output of training examples to improve the robustness of the trained model against noisy or mislabeled training data.

[0088] For example, the system can select one or more initial ground truth bounding boxes within a specific initial training image from a particular set of initial training examples, and then generate corresponding ground truth bounding boxes for the corresponding training examples by applying noise to the initial ground truth bounding boxes within the specific training image with respect to each selected initial ground truth bounding box. For example, the system can apply random scaling to the initial ground truth bounding boxes, apply random shifts to the initial ground truth bounding boxes, or both.

[0089] For each training image, the system generates a target output sequence (step 404) that includes, for each ground truth bounding box, a subsequence containing (i) a first set of tokens defining the position of the bounding box in the image, and (ii) a second token representing the ground truth object category of the bounding box. In other words, the target output sequence consists of one or more subsequences, each corresponding to a different ground truth bounding box. Optionally, the last subsequence of the target output sequence may be followed by a sequence end token, which is the last token in the target output sequence.

[0090] During inference, the order in which bounding boxes are arranged within the output sequence generated by the neural network is irrelevant; that is, an output that identifies bounding box A before bounding box B in the sequence should be considered equivalent to an output that places bounding box B earlier in the sequence. Therefore, the system can generate a target output sequence that reflects this. In particular, the system can order each subsequence within the target output sequence in a random order.

[0091] In some implementations, the system may also make one or more modifications to the target output sequence for any given training image in order to improve the performance of the neural network after training. As mentioned above, when a neural network is trained using "teacher forcing," the system also modifies the corresponding input sequence provided to the decoder neural network during training. Modifying the target output sequence (and optionally the corresponding input sequence) is called "sequence augmentation."

[0092] For example, as mentioned above, in some implementations, the token vocabulary may include tokens representing "noise" object categories that are not in the set of object categories. In these examples, the system may generate one or more random bounding boxes in the training image and, for each random bounding box, include a corresponding "noise" subsequence in the target output sequence, which includes (i) a first set of tokens defining the location of the random bounding box in the training image and (ii) a second set of tokens representing noise object categories that are not in the set of object categories.

[0093] As a specific example, the system can add a noise subsequence after the last subsequence corresponding to one of the ground truth bounding boxes.

[0094] In some implementations, the system may add a fixed number of noise subsequences to each target output sequence. In some other implementations, the system may add noise subsequences such that each target output sequence contains the same fixed number of subsequences (i.e., the same number of ground truth + noise subsequences).

[0095] Adding these random (or "noisy") bounding boxes to the target sequence can improve the performance of a trained neural network in various ways. For example, some of these noisy objects may be identical to or overlap with some of the ground truth objects, simulating noisy and overlapping predictions, i.e., training the neural network to be more robust to these types of predictions in the training data. As another example, introducing noisy bounding boxes to the target output sequence can prevent the neural network from ending up without identifying all objects in the image, without introducing noisy and overlapping predictions in the output produced as inference (as can happen when some other techniques are used, such as artificially reducing the scores assigned to EOS tokens).

[0096] The system then trains an object detection neural network to maximize the log-likelihood of any preceding token in the target output sequence and the training image for each training image and for each token in at least a subset of the tokens in the target output sequence for the training image (step 406).

[0097] When the target output sequence does not contain noisy bounding boxes, the above "at least a subset" may include all tokens in the target output sequence. When the target output sequence certainly contains noisy bounding boxes, the neural network is not trained to maximize the log-likelihood of tokens for a first set of tokens relating to random bounding boxes (but it is trained to maximize the log-likelihood of tokens for a second set of tokens representing the noise category). In other words, the above "at least a subset" includes all tokens except for the first set of tokens relating to random bounding boxes. This allows the neural network to learn to identify noisy bounding boxes rather than mimic them.

[0098] To train a neural network to maximize the log-likelihood, the system calculates the gradient of an objective function that measures the log-likelihood of at least a subset of tokens with respect to the parameters of the encoder and decoder neural networks by backpropagation, and then updates the parameters using the determined gradient. For example, the system can apply an appropriate optimizer to the gradient and parameters to update the parameters, e.g., an Adam optimizer, an rmsProp optimizer, an Adafactor optimizer, or a different machine learning optimizer. For example, the loss function could be, for each training example, the average of the combinations of log-likelihoods of the training output sequences of the training example, e.g., the sum or weighted sum. When the target output sequence certainly contains noisy bounding boxes, the first set of tokens with respect to random bounding boxes is not included in the combination.

[0099] Figure 5 shows an example of training a neural network with respect to a given target output sequence.

[0100] In particular, Figure 5 shows a first example 510 in which a neural network is trained without sequence augmentation, and a second example 512 in which a neural network is trained with sequence augmentation. More specifically, the first example 510 shows a target output sequence 520 and a corresponding input sequence 530. During training, the training system uses “supervisor forced” and therefore the neural network processes the input sequence 530 to generate the respective score distribution for each position in the target output sequence 520 used to train the neural network, as described above. Due to the “causality” employed by the decoder, the corresponding input sequence 530 is shifted by one token relative to the target output sequence 520, and therefore the respective score distribution for each given position in the target output sequence 520 depends on the tokens of the positions preceding the given position in the target output sequence 520 (and always the initial “start” token provided as the first input at the first time step during autoregressive generation at inference). This dependency is shown by arrows in Examples 510 and 512, where each token in the target output sequence depends on the corresponding input sequence, i.e., each token in the corresponding input sequence processed by the autoregressive decoder at each generation time step, which is connected to the token by the arrow.

[0101] In particular, in the first example 510, the target output sequence 520 includes 10 tokens y1 to y10 followed by an EOS token ("End"), and has no additional tokens.

[0102] The second example 512 also includes a target output sequence 540 and a corresponding input sequence 550. However, in the target output sequence 540, token y10 is followed by tokens relating to two “noisy” bounding boxes. Specifically, the target output sequence 540 includes two first sets of tokens representing the coordinates of the noise bounding boxes, each of which is followed by a “noise” token indicating that the bounding box is a noise bounding box that was not originally present in the corresponding input image. Similarly, the corresponding input sequence 550 also includes additional tokens following token y10. The first sets of tokens relating to the noisy bounding boxes are labeled “n / a” in Figure 5 because their log-likelihoods are not considered when training the neural network, for example, their losses are set to zero.

[0103] This specification uses the term “configured” in relation to systems and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action while in operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that cause a data processing device to perform the operation or action when executed by the device.

[0104] The embodiments and functional operations described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. The embodiments described herein can be implemented as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a receiver device suitable for execution by a data processing device.

[0105] The term “data processing device” refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, one programmable processor, one computer, or multiple processors or computers. A device may also be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include such dedicated logic circuits. Optionally, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code comprising one or more of these.

[0106] Computer programs, sometimes called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use within a computing environment. A program may, but not necessarily, correspond to a file in a file system. A program may be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple organized files, for example, one or more modules, subprograms, or files that store parts of code. A computer program can be deployed to run on one computer, located in one place, or distributed across multiple computers connected to each other by a data communication network.

[0107] In this specification, the term “database” is used broadly to refer to any collection of data, that is, data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on a storage device in one or more locations. Thus, for example, an index database can contain multiple collections of data, each of which may be organized and accessed in a different way.

[0108] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same one or more computers.

[0109] The processes and logic flows described herein may be executed by one or more programmable computers running one or more computer programs that perform functions by performing calculations on input data and generating outputs. Alternatively, the processes and logic flows may be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0110] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or incorporated into dedicated logic circuits. Generally, a computer includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is coupled to operate in a manner that it can receive data from or transfer data to such mass storage devices, or both. However, a computer does not have to have such devices. Furthermore, computers can be integrated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices, such as Universal Serial Bus (USB) flash drives.

[0111] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0112] To provide user interaction, the embodiments described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, on which the user can provide input to the computer. Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to the user's device's web browser in response to a request received from a web browser. The computer may also interact with the user by sending text messages or other forms of messages to a personal device running a messaging application, such as a smartphone, and receiving a response message from the user in return.

[0113] Data processing devices for implementing machine learning models may include, for example, dedicated hardware accelerator units for handling the computationally intensive parts of the workload, such as training or generating machine learning models or inference.

[0114] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0115] The embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a data server as a backend component, or a middleware component, for example, an application server, or a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with the subject implementation described herein, or one or more such backend components, middleware components, or frontend components in any combination. The components of the system can be interconnected by digital data communication in any form or medium, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0116] A computing system may include a client and a server. The client and server are generally geographically separated and typically interact through a communication network. The relationship between the client and the server arises from computer programs running on each computer that are in a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to the user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from such a user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0117] This specification includes many specific implementation details, but these should not be considered as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented in multiple embodiments separately or in any preferred partial combination. Furthermore, features may be described above as working in a particular combination, and may even be initially claimed as such, but one or more features of a claimed combination may, in some cases, be removed from the combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0118] Similarly, although the operations are shown in the figures and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in the specific order shown or sequentially, or that all shown operations be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the embodiments described above should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0119] A specific embodiment of the subject has been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process depicted in the appended diagrams does not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0120] 100 Object Detection Systems 102 Input Images 110 Object Detection Neural Networks 112 Output Sequence 120 Encoder Neural Networks 122 Encoded representation 130 Decoder Neural Network 150 Object Detection Output 200 processes 310 Input Images 312 Object detection output section 314 Object detection output section 316 Object detection output section 320 input images 330 Input Images 400 processes 510 Example 1 512 Second Example 520 Target Output Sequence 530 Input Sequence 540 Target Output Sequence 550 Corresponding input sequence

Claims

1. A method performed by one or more computers, Steps include obtaining the input image, A step of processing an input image using an object detection neural network to generate an output sequence containing each token in each of a plurality of time steps, wherein each token is selected from a vocabulary of tokens comprising (i) a first set of tokens each representing a discrete number from a set of discretized numbers, and (ii) a second set of tokens each representing an object category from a set of object categories, A method comprising the steps of generating data from the tokens in the output sequence, which includes one or more bounding boxes in the input image, and, for each bounding box, data that identifies each object category from the set of object categories to which the object depicted in the bounding box belongs.

2. The output sequence includes a subsequence corresponding to each of the one or more bounding boxes, and the step of generating the data that identifies the one or more bounding boxes is performed with respect to each bounding box. Identifying the coordinates of the bounding box in the input image from the tokens in the corresponding subsequence belonging to the first set of tokens, The method according to claim 1, further comprising identifying the object category to which each object depicted within the bounding box belongs, represented by the token in the corresponding subsequence, which belongs to the second set of tokens.

3. The method according to claim 2, wherein each of the subsequences comprises four tokens from the first set of tokens, and the four discrete numbers represented by the four tokens specify the coordinates of two corners of the bounding box in the input image.

4. The method according to claim 2, wherein each of the subsequences comprises four tokens from the first set of tokens, and four discrete numbers represented by the four tokens specify the coordinates of the center of the bounding box in the input image, and the height and width of the bounding box.

5. The step of processing the input image using the object detection neural network is: The input image is processed using an encoder neural network to generate an encoded representation of the input image, The method according to any one of claims 1 to 4, further comprising processing the encoded representation of the input image using a decoder neural network to generate the output sequence.

6. The object detection neural network is configured to generate a score distribution on the tokens of the vocabulary for each time step, conditional on (i) the input image and (ii) the tokens in any previous time step in the output sequence, and the step of processing the input image using the object detection neural network to generate the output sequence is, with respect to each time step, The method according to any one of claims 1 to 5, comprising selecting each of the tokens in the time step within the output sequence using the respective score distributions generated by the object detection neural network with respect to the time step.

7. The method according to claim 6, wherein selecting each of the aforementioned tokens includes selecting the token having the highest score in each of the aforementioned score distributions.

8. The method according to claim 6, wherein selecting each of the aforementioned tokens includes sampling tokens according to the score distribution.

9. The method according to claim 8, wherein selecting each of the aforementioned tokens includes sampling the tokens according to the score distribution using nuclear sampling.

10. The vocabulary includes noise tokens representing noise categories not included in the set of object categories, and the step of processing the input image using the object detection neural network to generate an output sequence is, with respect to a particular time step of the time steps, Determining that the token with the highest score for the aforementioned specific time step is the noise token, The method according to any one of claims 6 to 9, further comprising, in response, selecting the token having the highest score from only the tokens of the second set of tokens.

11. The decoder neural network, with respect to each time step, The method according to any one of claims 6 to 10, when dependent on claim 5, which is configured to process the tokens in any previous time step in the output sequence, subject to an encoded representation of the input image in order to generate a score distribution for each of the time steps.

12. The method according to any one of claims 6 to 11, further comprising the step of associating, with respect to each of the one or more bounding boxes, the respective scores assigned to the tokens representing the respective object categories of the bounding boxes in the score distribution at a corresponding time step, in order to represent the degree of confidence that the respective object category is the correct category of the object.

13. The method according to any one of claims 1 to 12, further comprising the step of outputting data that identifies one or more bounding boxes in the input image and, with respect to each bounding box, the respective object category from the set of object categories to which the object drawn within the bounding box belongs.

14. A method for training an object detection neural network according to any one of claims 1 to 13, Steps include obtaining a batch of training images, and for each training image, obtaining a target output that identifies one or more ground truth bounding boxes in the image and the respective ground truth object category of each bounding box, For each training image and for each ground truth bounding box, the step of generating a target output sequence includes, for each training image and each ground truth bounding box, a subsequence comprising (i) a first set of tokens defining the position of the bounding box in the image, and (ii) a second set of tokens representing the ground truth object category of the bounding box. A method comprising the step of training the object detection neural network to maximize the log-likelihood of a token given any preceding token in the target output sequence and the training image, with respect to each training image and for each token in at least a subset of tokens in the target output sequence with respect to the training image.

15. The steps include obtaining a batch of training images, and for each training image, obtaining a target output that identifies one or more ground truth bounding boxes in the image and the respective ground truth object category of each bounding box, The method according to claim 14, comprising generating one or more of the training images of the batch by applying one or more image enhancement policies to the corresponding initial training images.

16. The steps include obtaining a batch of training images, and for each training image, obtaining a target output that identifies one or more ground truth bounding boxes in the image and the respective ground truth object category of each bounding box, The method according to claim 14 or 15, comprising generating a bounding box with respect to a specific bounding box in a specific training image by applying noise to an initial ground truth bounding box in the specific training image.

17. For each training image, the step of generating the target output sequence is: To generate one or more random bounding boxes within the aforementioned training image, The method according to any one of claims 14 to 16, wherein, for each random bounding box, the target output sequence includes (i) a first set of tokens defining the location of the random bounding box in the training image, and (ii) a second token representing a noise object category not present in the set of object categories.

18. The method according to claim 17, wherein the object detection neural network is not trained to maximize the log-likelihood of the tokens in the set of first tokens relating to the random bounding box.

19. For each training image, the step of generating the target output sequence is: The method according to any one of claims 14 to 18, comprising ordering each of the subsequences in a random order within the target output sequence.

20. One or more computers, A system including one or more storage devices that store instructions causing one or more computers to perform the method according to any one of claims 1 to 19 when executed by the one or more computers.

21. One or more computer-readable storage media for storing instructions causing one or more computers to perform each of the operations described in any one of claims 1 to 19 when executed by one or more computers.

Images