Performing computer vision task by generating sequence of lexical elements

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a plurality of computer vision tasks using a shared computer vision neural network. In one aspect, a method in the aspect includes obtaining an input image; processing the input image and the hint sequence using a shared computer vision neural network to generate an output sequence comprising a respective lexical element at each of a plurality of time steps, where each lexical element is selected from a shared lexical element vocabulary shared between a plurality of computer vision tasks, wherein the shared vocabulary comprises (i) a first set of lexical elements, each representing a respective discrete number from the set of discretized numbers, and (ii) a second set of lexical elements, each representing a natural language text lexical element.

Description

Background Art

[0001] This specification relates to using neural networks to process images.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input of the next layer in the network (i.e., another hidden layer or the output layer). Each layer of the network generates an output from the received input according to the current values ​​of the corresponding set of parameters. Summary of the invention

[0003] This specification describes a system implemented as a computer program on one or more computers in one or more locations that uses a shared computer vision neural network to perform any of a plurality of computer vision tasks.

[0004] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0005] Conventionally, the inputs and outputs of various computer vision tasks have been represented in very different output spaces and sometimes in different input spaces. As a result, there has been a proliferation of different architectures and loss functions for different vision tasks.

[0006] On the other hand, using the described techniques, a single shared computer vision neural network can be used to perform multiple different computer vision tasks. In particular, by representing the output of each task as a sequence of tokens, the shared computer vision neural network can be used to perform tasks with different output spaces. As a result, the described system consumes significantly less memory than other single-task approaches while providing significantly better accuracy on multiple tasks than other multi-task approaches. That is, by formulating the output of each task as a sequence of discrete tokens using a unified interface, a neural network with a single model architecture and loss function can be trained on all of these tasks simultaneously without task-specific customization, while still achieving comparable or better performance on multiple tasks than dedicated architectures.

[0007] Thus, the described system consumes significantly less memory when deployed to perform multiple tasks relative to a system that requires a separate neural network for each of the multiple tasks. That is, the described system only requires storing a single set of network parameters for a shared computer vision neural network, rather than multiple sets of network parameters for multiple different task-specific computer vision neural networks.

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0009] Other features, aspects, and advantages of the subject matter will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of an example multi-task computer vision system.

[0011] Figure 2 is a flowchart of an example process for performing a computer vision task.

[0012] Figure 3 is a diagram showing examples of task outputs generated using a shared computer vision neural network.

[0013] Figure 4 is a diagram illustrating the training of a shared computer vision neural network.

[0014] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0015] Figure 1 is a diagram of an example multi-task computer vision system 100. The multi-task computer vision system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0016] The multi-task computer vision system 100 is a system that receives an input image 102 and generates a task output 150 for the input image 102 by processing the image 102 using a shared computer vision neural network 110 to generate a task output 150 for a computer vision task.

[0017] The system 100 is referred to as a “multi-tasking” computer vision system 100 because the system 100 is configured to perform any of a plurality of computer vision tasks.

[0018] Similarly, the shared computer vision neural network 110 is referred to as “shared” because the system uses the same computer vision neural network 110 to perform all of the multiple tasks.

[0019] The system may be configured to perform two or more of a variety of computer vision tasks. Typically, a computer vision task is a task that requires processing an image (ie, processing the intensity values ​​of pixels of the image) to generate a prediction that characterizes the image.

[0020] What now follow are some non-limiting examples of computer vision tasks that system 100 may be configured to perform.

[0021] In some examples, the computer vision task includes one or more tasks that require making predictions that characterize objects depicted in an input image. For example, the computer vision task may include one or more of: object detection of one or more objects depicted in the input image; object keypoint detection; object localization; object instance segmentation; and / or object description (captioning).

[0022] As an example, one of the tasks may be object detection. For the object detection task, the output 150 identifies one or more bounding boxes in the input image 102, each of which corresponds to a detected object, i.e., a depiction predicted to contain the detected object, and for each of the bounding boxes, the output identifies an object category from a set of object categories to which the object detected in the bounding box belongs.

[0023] As another example, one of the tasks may be a keypoint detection task. For keypoint detection, the output 150 specifies the coordinates of one or more keypoints on one or more objects depicted in the input image 102 .

[0024] As yet another example, one of the tasks may be an image captioning task. For image captioning, the output 150 is a text sequence that captions the input image 102 , ie, a text caption describing the content of the input image 102 .

[0025] As yet another example, one of the tasks may be an instance segmentation task. For instance segmentation, the output 150 identifies instances of objects within the input image 102 for one or more objects in the input image. That is, the instance segmentation output identifies which pixels in the image belong to each of the one or more instances of the object depicted in the input image.

[0026] More specifically, the system 100 obtains an input image 102 and task data 104 that specifies a target computer vision task from among a plurality of computer vision tasks to be performed on the input image 102 .

[0027] The system generates a prompt sequence 106 from the task data 104 that describes a target computer vision task (ie, describes the computer vision task specified by the task data 104 ).

[0028] The system 100 uses a shared computer vision neural network 110 to process an input image 102 (ie, process intensity values ​​of pixels of the input image 102) and a prompt sequence 106 to generate an output sequence 112 including a plurality of word-grams.

[0029] Each word-gram is selected from a shared word-gram vocabulary shared between multiple computer vision tasks. The shared vocabulary includes (i) a first set of word-grams, each representing a corresponding discrete number from a discretized set of numbers (i.e., a set of numbers that can be mapped to quantized image coordinates in an input image), and (ii) a second set of word-grams, each representing a natural language text word-gram. The natural language text word-grams may include (i) category labels for a set of object categories and (ii) other natural language text word-grams that can be used to generate natural language text, for example, word-grams representing any of the following: words, characters, subwords or word fragments, punctuation marks, separator characters, etc.

[0030] Typically, the prompt sequence 106 is also composed of tokens selected from the shared vocabulary, so that the prompt sequence 106 can be processed by the shared computer vision neural network 110. The prompt sequence 106 may include a natural language description of the task, and optionally, may specify one or more parameters required to perform the task, such as a bounding box representing a portion of the image for which the task is to be performed.

[0031] Word-grams in the vocabulary may be represented in any suitable manner, for example, as integers or other alphanumeric symbols that are distinguishable from one another.

[0032] More specifically, the system discretizes the numbers used to represent the coordinates of any given point in the input image 102 into a plurality of bins, where each of the bins corresponds to a respective one of the word-grams in the first set of word-grams. As a specific example, the system may discretize the range of values ​​between zero and the height or width in pixels in the image into a fixed number of evenly spaced bins, such that each bin corresponds to a different subset of pixel indices that can be used to represent the coordinates of a point in the image (in pixels). Thus, if the input image is 600 pixels by 600 pixels and there are 600 bins, each bin will correspond to a different pixel index from 1 to 600. If the input image is 600x600 and there are 300 bins, each bin will correspond to a different set of two pixel indices from 1 to 600.

[0033] Thus, each word-gram in the first word-gram set represents a different interval in the discretization of possible coordinate values ​​for pixels in the image, and can be mapped to a different quantized coordinate value, for example, a representative value of the interval represented by the word-gram. For example, the representative value can be the average of the endpoints of the interval, or one of the two endpoints of the interval. This quantization scheme for coordinates allows the system to use a relatively small vocabulary to represent possible pixel coordinates while maintaining high accuracy.

[0034] Thus, as a specific example, when there are 600 intervals in the quantization scheme and word-grams are represented as integers, the vocabulary may include word-grams 1-600 representing the 600 possible quantization coordinates and word-grams 601 and above representing natural language text word-grams.

[0035] Optionally, as will be described in more detail, the vocabulary may include one or more additional word-grams in addition to those described above.

[0036] Typically, the shared computer vision neural network 110 is configured to generate an output sequence across multiple time steps.

[0037] At each time step, the neural network 110 is configured to generate a distribution of scores over the word-grams in the vocabulary for each time step, conditioned on (i) the input image and (ii) the word-grams at any earlier time step in the output sequence.

[0038] Thus, at each time step during generation of the output sequence 112 , the system 100 selects a corresponding word-gram at that time step in the output sequence 112 using the corresponding score distribution generated by the shared computer vision neural network 110 for that time step.

[0039] As one example, the system 100 may greedily select the highest scoring word-gram.

[0040] As another example, the system 100 can select the corresponding word-gram by sampling the word-grams according to a score distribution.As a specific example, the system can sample the word-grams according to a score distribution using kernel sampling.

[0041] As a specific example, the shared computer vision neural network 110 may include an encoder neural network 120 and a decoder neural network 130 .

[0042] The encoder neural network 120 may be configured to process the input image 102 to generate an encoded representation 122 of the input image 102. The encoded representation 122 is a sequence including a plurality of encoding vectors that collectively represent the input image 102.

[0043] The encoder neural network 120 may be any suitable image encoder neural network that receives intensity values ​​of pixels of the image 102 and encodes the intensity values ​​into a hidden representation. Examples of such encoders include convolutional neural networks, Transformer neural networks, or neural networks that include both convolutional layers and self-attention layers. Examples of convolutional neural networks that can be used as encoders are described in: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. An example of a Transformer neural network that can be used as an encoder is described in: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al., An image is worth 16x16words: Transformers for image recognition at scale, in International Conference on Learning Representations, 2020.An example of a neural network that includes both convolutional layers and self-attention layers that can be used as an encoder is described in: Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, End-to-end object detection with transformers, in European Conference on Computer Vision, pp. 213-229, Springer, 2020.

[0044] When the last layer of the encoder 120 is a convolutional layer that generates a feature map, the system 100 can generate the encoded representation 122 by flattening the feature map into a vector sequence. When the last layer of the encoder 120 is an attention layer, the system 100 can directly use the output of the attention layer as the encoded representation 122.

[0045] The decoder neural network 130 is configured to process the encoded representation 122 of the input image 102 and the prompt sequence 106 to generate the output sequence 112 .

[0046] In particular, the decoder 130 may be an autoregressive decoder neural network that processes the prompt sequence 106 at each time step and then processes any token at any earlier time step in the output sequence 112, while conditioned on the encoded representation 122 of the input image 102, to generate a corresponding score distribution for that time step. The score distribution includes a corresponding score, e.g., a probability or log-odds, for each token in the vocabulary.

[0047] As a specific example, the decoder 130 may be a Transformer decoder that applies causal self-attention to the prompt sequence 106 and the already generated tokens, and applies criss-cross attention to the encoded representation 122. That is, the decoder 130 may include both a self-attention layer that applies causal self-attention to the representation of the prompt sequence and the already generated tokens, and a criss-cross attention layer that criss-cross-attentions the encoded representation 122.

[0048] Examples of such Transformer decoders that can be used as decoder 130 are described in: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint arXiv:1910.10683, 2019, and Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, Improving language understanding by generative pre-training, 2018.

[0049] Then, the system 100 generates a task output 150 of the target computer vision task from the word-grams in the output sequence 112. That is, the output of each task in the task can be represented by a sequence of word-grams in the shared vocabulary. For a given task, the system 100 maps the word-grams in the output sequence 112 to the output of the task according to the predetermined grammar of the task.

[0050] As a specific example, for object detection, the system 100 maps word-grams in the output sequence 112 to data identifying one or more bounding boxes in the input image, and for each bounding box, the data identifies a corresponding object category from a set of object categories to which the object depicted within the bounding box belongs.

[0051] For each bounding box, the data identifying the bounding box specifies the location of the bounding box within the image. As an example, the data identifying the bounding box may specify the coordinates of two or more of the corners of the bounding box. As another example, the data identifying the bounding box may specify the coordinates of the center of the bounding box and the height and width of the bounding box.

[0052] The syntax and examples of mappings for various computer vision tasks are described in more detail below.

[0053] Thus, the shared computer vision neural network 110 generates an output sequence 112 of discrete tokens that can be directly mapped by the system 100 to the output of a computer vision task. The neural network 110 generates outputs for different tasks in the task by conditioning on different prompt sequences 106. By generating task outputs in this manner, the system 100 does not require a high degree of customization or use of neural networks with complex architectures, and can be easily integrated into larger systems. Furthermore, by generating task outputs, the system 100 generates a unified interface that allows the same model to be used for all of the tasks in the multiple tasks without requiring a customized architecture for any of the tasks.

[0054] As a specific example, system 100 may be part of a perception system embedded within an agent (e.g., a robot or an autonomous vehicle) that processes images and optionally other sensor data collected by the agent's sensors, and task outputs may be used by the perception system or other software onboard the agent to control the agent as it navigates through an environment.

[0055] As another specific example, the system 100 may be part of a perception system embedded within or in communication with different types of devices that process sensor data, such as camera surveillance systems, mobile phones, etc. The task outputs generated by the system 100 may be used as part of a pre-processing stage before displaying an image to a user, or may be used to automatically trigger other actions.

[0056] As yet another specific example, a client device may interact with the system 100 via an application programming interface (API) (e.g., a web-based API). In particular, the client device may submit an API call that includes or identifies an image to be analyzed and one or more tasks to be performed on the image. In response, the system 100 may provide data identifying task outputs. For example, the system 100 may format the task outputs in a specified format, such as a JavaScript Object Notation (JSON) file or a file in another type of data exchange format, and provide the file in response to the API call.

[0057] Before using neural network 110 to perform these tasks, system 100 or another training system trains neural network 110 on training data including a plurality of training examples.

[0058] Each training example includes an input training image and a ground-truth output for that image for one or more of the computer vision tasks.

[0059] The following will refer to Figure 4 Training the neural network 110 is described in more detail.

[0060] Figure 2 is a flow chart of an example process 200 for generating task outputs for an input image. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a multi-task computer vision system (e.g., Figure 1 The multi-task computer vision system 100 depicted in FIG. 1 may perform process 200 .

[0061] The system obtains an input image (step 202).

[0062] The system obtains data specifying a target computer vision task from among a plurality of computer vision tasks to be performed on an input image (step 204).

[0063] The system generates a prompt sequence that describes the target computer vision task (step 206).

[0064] Typically, the system generates a prompt sequence that includes word-grams from a shared vocabulary and identifies a target task to a computer vision neural network.

[0065] The following will refer to Figure 3 Examples of prompt sequences describing various tasks.

[0066] The system processes the input image and prompt sequence using a shared computer vision neural network to generate an output sequence (step 208).

[0067] As described above, the output sequence includes a corresponding word-gram at each time step in a plurality of time steps. Each word-gram is selected from a shared word-gram vocabulary shared between a plurality of computer vision tasks. The shared vocabulary includes (i) a first set of word-grams, each representing a corresponding discrete number from a discretized set of numbers (i.e., a set of numbers that can be mapped to quantized image coordinates in an input image), and (ii) a second set of word-grams, each representing a natural language text word-gram.

[0068] The vocabulary may also optionally include additional word-grams.

[0069] For example, the vocabulary may include end-of-sequence (EOS) tokens.

[0070] Typically, as described above, a shared computer vision neural network is configured to generate an output sequence across multiple time steps.

[0071] At each time step, the neural network is configured to generate a distribution over scores over the tokens in the vocabulary for each time step, conditioned on (i) the input image, (ii) the prompt sequence, and (ii) the token at any earlier time step in the output sequence.

[0072] Thus, at each time step during generation of the output sequence, the system uses the corresponding score distribution generated by the shared computer vision neural network for that time step to select the corresponding word-gram at that time step in the output sequence.

[0073] As an example, the system may greedily select the highest scoring word-gram.

[0074] As another example, the system can select the corresponding word-gram by sampling the word-grams according to a score distribution.As a specific example, the system can sample the word-grams according to a score distribution using kernel sampling.

[0075] In some implementations, the system continues to add tokens to the output sequence until an end-of-sequence (EOS) token is selected. In some other implementations, the system continues to add tokens to the output sequence until the output sequence has a fixed length, i.e., has a maximum number of tokens. In still other implementations, the system continues to add tokens to the output sequence until an EOS token has been selected, or until the output sequence has a fixed length, whichever occurs first.

[0076] The system generates a task output for the machine learning task from the tokens in the output sequence (step 206).

[0077] In particular, because of the way the neural network is trained, since a sequence of prompts is provided as input to the neural network, the output sequence will conform to the grammar of the target computer vision task.

[0078] As a specific example, for object detection, the task output requires data identifying one or more bounding boxes in the input image, and for each bounding box, the data identifying a corresponding object category from a set of object categories to which the object depicted in the bounding box belongs.

[0079] When the target computer vision task is object detection, the output sequence includes a corresponding subsequence of each of the one or more bounding boxes in the input image.

[0080] A subsequence of a given bounding box includes word-grams from a first word-gram set and word-grams from a second word-gram set. For example, the subsequence may include a total of five word-grams: four word-grams from the first set and one word-gram from the second set. As a specific example, the subsequence may include four word-grams from the first set followed by one word-gram from the second set.

[0081] In some cases, the four discrete numbers represented by the four tokens from the first set specify the coordinates of two corners of the bounding box in the input image, e.g., the (x, y) coordinates of the upper left and lower right corners or the lower right and upper left corners.

[0082] In some other cases, the four discrete numbers represented by the four tokens from the first set specify the coordinates of the center of the bounding box in the input image and the height and width of the bounding box.

[0083] Thus, to generate object detection output, the system may, for each subsequence in the output sequence, identify the coordinates of a bounding box in the input image from the tokens in the subsequence that belong to the first token set, and then identify the object category represented by the tokens in the corresponding subsequence that belong to the second token set as the corresponding object category to which the object depicted in the bounding box belongs.

[0084] In other words, for each first word-gram in the subsequence, the system may map the first word-gram to a quantized coordinate value represented by the word-gram to generate coordinates that define a bounding box, and for each second word-gram, the system may map the word-gram to a label or other data identifying the object category represented by the second word-gram.

[0085] In some implementations, the system also associates with the bounding box a respective score of a word-gram in the score distribution at the corresponding time step assigned to the respective object category representing the bounding box to indicate a confidence that the respective object category is the correct category for the object.

[0086] In some implementations, the system may avoid including one or more bounding boxes in the object detection output, for example, if the confidence score for the corresponding object class is below a threshold.

[0087] When the task is a keypoint prediction task, the prompt sequence identifies object instances in the input image, i.e., identifies objects on whose surfaces keypoints should be identified.

[0088] The output sequence includes a corresponding subsequence of quantized image coordinate values ​​for each of the one or more keypoints, the quantized image coordinate values ​​specifying a position of the keypoint in the input image. That is, the subsequence includes two word-grams from the first set for each of the one or more keypoints, where the two word-grams represent the (x, y) coordinates of the keypoint in the input image. The system can then map the two word-grams to generate a dequantized (x, y) coordinate of the keypoint.

[0089] When the task is image captioning, the output sequence is a sequence of tokens from the second token set, which tokens represent the textual caption of the input image. For this task, the system can generate a textual sequence directly from the sequence of tokens, and output the textual sequence as the task output, i.e., the textual caption of the input image.

[0090] When the task is instance segmentation, the prompt sequence will identify an instance of an object. For this task, the output sequence is a sequence of tokens from the first token set that represent quantized coordinates of a polygon overlaying the object instance in the input image. The system can then parse the sequence of tokens to map the quantized coordinates to image coordinates, and then generate a dense mask on the image that specifies pixels belonging to the object instance, i.e., such that pixels inside the polygon are identified as belonging to the object instance, and pixels outside the polygon are identified as not belonging to the object instance.

[0091] In some cases, for instance segmentation tasks, to improve performance, the system may generate one or more additional output sequences conditioned on the same prompt sequence so as to generate a corresponding dense mask for each of the additional output sequences. The system may then average the dense masks of the additional output sequences and the dense masks of the original output sequences to generate a final dense mask for generating the final task output. For example, the system may apply a threshold to the averaged mask to obtain a single binary mask, and use the binary mask as the task output.

[0092] The system may then output data identifying the output of the task.

[0093] As a specific example, when the system is part of a perception system embedded within an intelligent agent (e.g., a robot or autonomous vehicle) that processes images and optionally other sensor data collected by the agent's sensors, the system may provide data identifying task outputs to the perception system or other software onboard the agent to control the agent as it navigates through an environment.

[0094] As another specific example, when the system is part of a perception system embedded within or communicating with a different type of device that processes sensor data (e.g., a camera surveillance system, a mobile phone, etc.), the system may output the data to another software component of the device for use in pre-processing the image before displaying it to a user, or for automatically triggering an action, such as an alarm.

[0095] As yet another specific example, when a client device can interact with the system via an application programming interface (API) (e.g., a web-based API), the system can provide data identifying the task output in response to the API call. For example, the system can format the object detection output in a specified format, such as a JavaScript Object Notation (JSON) file or a file in another type of data exchange format, and provide the file in response to the API call.

[0096] Figure 3 Example task outputs extracted from the output sequences generated by a shared computer vision neural network are shown.

[0097] In particular, Figure 3 Portions of example task outputs for four example input images 302 , 304 , 306 , and 308 are shown.

[0098] from Figure 3 It can be seen that the system uses the neural network 110 to process each of the input images 302, 304, 306 and 308 and the corresponding prompt sequence to generate a corresponding output sequence, and then extracts the corresponding task output parts 312, 314, 316 and 318 of the task specified by the corresponding prompt sequence from the output sequence for the input image.

[0099] Typically, the prompt sequence for each computer vision task includes one or more tokens that uniquely identify the computer vision task (ie, are different for each computer vision task). For example, these tokens may be a natural language description of the task.

[0100] from Figure 3 As can be seen, the prompt for the object detection task (ie, the task performed on the input image 302) includes a natural language phrase identifying the task, eg, “[detect]”.

[0101] exist Figure 3 In the example of , the object detection output portion 312 for image 302 specifies that the bounding box has a lower right corner at y_min = 327 and x_min = 370 (in pixel coordinates), an upper right corner at y_max = 653 and x_max = 44, and is an image of an object belonging to the "train" category.

[0102] For example, when word-grams are represented as integers, the image is 400x400, and each first word-gram corresponds to one pixel of the image, the system may extract the object detection output portion from the subsequence [327, 370, 653, 44, 415], where "415" is a word-gram representing the "train" category. That is, although portion 312 is shown as including identification information for each element, the underlying output sequence is simply a sequence of discrete word-grams from the vocabulary. Alternatively, the vocabulary may include one or more text word-grams, such as words, word fragments, or characters, representing the "train" category, i.e., rather than integers that can be mapped to a category identifier.

[0103] from Figure 3As can be seen, the hint for the instance segmentation task (i.e., the task performed on the input image 304) includes a natural language phrase identifying the task (e.g., "[Segment]") and the quantized coordinates of the bounding box containing the object to be segmented. As with portion 312, although the hint is shown to include identification information for each element, the underlying hint sequence may only include a sequence of discrete tokens from a vocabulary.

[0104] exist Figure 3 In the example of , segmentation output portion 314 of image 304 specifies quantized coordinates of vertices of a polygon overlaid on image 304. By utilizing EOS tokens, neural network 110 can generate a sequence of coordinates representing a varying number of vertices in order to segment an object of arbitrary shape.

[0105] from Figure 3 As can be seen, the prompt for the keypoint identification task (i.e., the task performed on the input image 306) includes a natural language phrase for the identification task (e.g., "[Keypoint]") and the quantized coordinates of the bounding box that contains the agent with the keypoint to be identified. As with portion 312, although the prompt is shown as including identification information for each element, the underlying prompt sequence may only include a sequence of discrete tokens from a vocabulary.

[0106] exist Figure 3 In the example of FIG. 306 , the keypoint output portion 316 for the image 316 specifies the quantized coordinates of the keypoints of the agent in the image 306. By utilizing EOS tokens, the neural network 110 can generate a sequence of coordinates representing a varying number of keypoints.

[0107] from Figure 3 As can be seen, the prompt for the captioned task (ie, the task to be performed on the input image 308) includes a natural language phrase identifying the task, eg, "[Describe]".

[0108] exist Figure 3 In the example of , portion 318 of the caption output for image 308 represents a natural language text sequence describing the content of image 308 (i.e., "A person working in mechanical shop with two mopeds outside"). For example, neural network 318 can generate outputs of word units representing word fragments that can be directly mapped to the natural language text sequence.

[0109] Therefore, from Figure 3It can be seen that by modifying the prompt sequence, the neural network 110 can be used to perform a number of different computer vision tasks, all of which have outputs that are representable as differently formatted output sequences (ie, output sequences arranged according to different grammars).

[0110] Figure 4 is a diagram 400 illustrating the training of neural network 110 .

[0111] As described above, before using neural network 110 to perform a plurality of tasks, system 100 or another training system trains neural network 110 on training data including a plurality of training examples.

[0112] Each training example includes an input training image 402 and a ground-truth output of the image for one or more of the computer vision tasks.

[0113] In particular, the system may repeatedly perform iterations of the training process on different batches of training examples to train the neural network, i.e., to repeatedly adjust the values ​​of the parameters of the neural network. That is, at each iteration of the training process, the system obtains a batch of one or more training examples (e.g., by sampling the batches from a larger training data set), and then performs iterations of the training process to update the current values ​​of the network parameters at the end of the iteration.

[0114] For example, the system may continue to perform iterations of the training process until a termination criterion has been met, eg, until a threshold number of training iterations have been performed, until a specified amount of time has elapsed, or until parameter convergence has been determined.

[0115] At each iteration of the training process, the system obtains a batch of training examples. Each training example includes a training image 402 and a target output of one of the computer vision tasks. In some implementations, the batch includes only training examples for one of the tasks, while in other implementations, the batch may include training examples for multiple tasks in the task.

[0116] In some implementations, the system applies one or more augmentation techniques to generate the batch from an initial batch of training examples.

[0117] As an example, the system can generate one or more training images in the batch of training images by applying one or more image enhancement strategies to the corresponding initial training images. The system can then associate each generated training image with a target output for the corresponding initial training image. Applying the image enhancement strategy can improve the robustness of the trained neural network to various image perturbations that may not be well represented in the training data.

[0118] For example, an image augmentation strategy may specify how random scaling, cropping, or other image augmentation techniques are applied to the initial training images to generate the batch of training examples.

[0119] As an example, the system can perform scale jittering and random cropping on the initial training images. Examples of such techniques are described in: Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D. Cubuk, Quoc V Le, and Barret Zoph, Simple copy-paste is a strong data augmentation method for instance segmentation, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2918–2928, 2021.

[0120] As another example, the system may perform image scaling (with a fixed aspect ratio) by resizing the image so that the longer side has a fixed number of pixels.

[0121] As another example, the system may apply color distortion to the intensity values ​​of pixels in the initial training image. Examples of such techniques are described in: Andrew G Howard, Some improvements on deep convolutional neural network based image classification, arXiv preprint arXiv:1312.5402, 2013.

[0122] As an alternative or in addition to image augmentation, the system can also apply augmentation to the target outputs in the training examples to increase the robustness of the trained model to noisy predictions or incorrectly labeled training data, such as by injecting noise into the target outputs.

[0123] For each training image X 图像 402, the system generates the target output sequence Y 构建 404, the sequence describes the real value target output in the syntax corresponding to the computer vision task. The target output sequence 404 also includes a prompt sequence corresponding to the task.

[0124] The system then trains the shared computer vision neural network 110 to maximize, for each training image and for each word-gram in at least a subset of word-grams in the target output sequence for the training image, the log-likelihood of the word-gram conditioned on any previous word-gram in the target output sequence and the training image.

[0125] To train the neural network to maximize the log-likelihood, the system computes by backpropagation the gradient of an objective function 408, which measures the log-likelihood of at least a subset of tokens, with respect to the parameters of the encoder neural network and the decoder neural network, and then uses the determined gradients to update the parameters. For example, the system may apply an appropriate optimizer (e.g., an Adam optimizer, an rmsProp optimizer, an Adafactor optimizer, or a different machine learning optimizer) to the gradients and parameters to update the parameters. For example, for each training example, the loss function may be the average of a combination (e.g., a sum or a weighted sum) of the log-likelihoods of the training output sequences in the training example. In other words, the target 408 may be a weighted next-term prediction target, where the weights of the prompt tokens in the prompt sequence are set to zero. For example, the system may use the neural network 110 to process Y using teacher forcing 构建 404, to obtain the likelihood Y for calculating the log-likelihood pred 406.

[0126] In some implementations, rather than processing a single batch at each training iteration, the system processes multiple batches, each batch corresponding to a different task in the multiple tasks. The system can then aggregate their gradients (e.g., using an average or weighted average), and then apply the optimizer to the aggregated gradients as described above.

[0127] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0128] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information to be sent to a suitable receiver device for execution by a data processing device.

[0129] The term "data processing equipment" refers to data processing hardware and encompasses all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or include a dedicated logic circuit system, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0130] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0131] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0132] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0133] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.

[0134] A computer suitable for executing a computer program may be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a dedicated logic circuit system. Typically, the computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or may be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, the computer does not need to have such a device. In addition, the computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0135] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.

[0136] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application), and receiving a response message from the user in return.

[0137] A data processing device used to implement a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing general-purpose and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0138] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).

[0139] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0140] A computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship of the client and the server is due to computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from it. Data generated at the user device can be received from the device at the server, for example, due to user interaction.

[0141] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0142] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0143] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or ordered order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: Get the input image; obtaining data specifying a target computer vision task from among a plurality of computer vision tasks to be performed on the input image; generating a prompt sequence describing the target computer vision task; processing the input image and the prompt sequence using a shared computer vision neural network shared among the plurality of computer vision tasks to generate an output sequence comprising a corresponding word-gram at each of a plurality of time steps, wherein each word-gram is selected from a shared word-gram vocabulary shared among the plurality of computer vision tasks, wherein the shared vocabulary comprises (i) a first set of word-grams, each word-gram representing a corresponding discrete number from a discretized set of numbers, and (ii) a second set of word-grams, each word-gram representing a natural language text word-gram; and An output of the target computer vision task is generated from the word-grams in the output sequence.

2. The method of claim 1, wherein the shared computer vision neural network comprises: an encoder neural network configured to process the input image to generate an encoded representation of the input image, and An autoregressive decoder neural network configured to autoregressively generate the output sequence conditioned on the encoded representation of the input image and the prompt sequence.

3. The method of claim 2, wherein the prompt sequence is a sequence of tokens from the shared vocabulary, and wherein the autoregressive decoder neural network is configured to, at each time step: An input sequence comprising (i) the prompt sequence and (ii) any word-grams at any earlier time step in the output sequence is processed to generate a probability distribution over the word-grams in the vocabulary.

4. The method of claim 2 or claim 3, wherein the autoregressive decoder neural network is an autoregressive self-attention decoder neural network.

5. A method according to any preceding claim, wherein the encoder neural network is a visual Transformer, a convolutional neural network, or a neural network comprising both a convolutional neural network layer and a self-attention layer.

6. A method according to any preceding claim, wherein the computer vision task is object detection, and wherein generating an output of the target computer vision task from the word-grams in the output sequence comprises: Data identifying one or more bounding boxes in the input image are generated from the word-grams in the output sequence, and for each bounding box, the data identifies a corresponding object category from the set of object categories to which an object depicted in the bounding box belongs.

7. The method of claim 6, wherein the output sequence comprises a respective subsequence corresponding to each of the one or more bounding boxes, and wherein generating the data identifying the one or more bounding boxes comprises, for each bounding box: Identify the coordinates of the bounding box in the input image from the word-grams belonging to the first word-gram set in the corresponding subsequence; and The object category represented by a set of one or more word-grams belonging to the second set of word-grams is identified as the corresponding object category to which the object depicted in the bounding box belongs.

8. The method of claim 7, wherein the corresponding subsequence comprises four word-grams from the first set of word-grams, and wherein four discrete numbers represented by the four word-grams specify coordinates of two corners of the bounding box in the input image.

9. The method of claim 7, wherein the corresponding subsequence comprises four word-grams from the first set of word-grams, and wherein four discrete numbers represented by the four word-grams specify coordinates of a center of the bounding box in the input image and a height and a width of the bounding box.

10. The method according to any one of claims 7 to 9, further comprising: A confidence score for the object is generated from respective scores assigned by the neural network to the set of one or more word-grams.

11. A method according to any one of claims 1 to 6, wherein the computer vision task is a keypoint prediction task, wherein the prompt sequence identifies object instances in the input image, and wherein the output sequence includes a corresponding subsequence of quantized image coordinate values ​​for each of one or more keypoints, wherein the quantized image coordinate values ​​specify the position of the keypoint in the input image.

12. The method of claim 11, wherein the corresponding subsequence of each key point comprises a set of word-grams from the second set of word-grams that represent a description of the key point.

13. The method according to any one of claims 1 to 6, wherein the computer vision task is image captioning, and wherein the output sequence is a sequence of word-grams from the second word-gram set representing a textual caption of the input image.

14. A method according to any one of claims 1 to 6, wherein the computer vision task is instance segmentation, wherein the prompt sequence identifies an instance of an object, and wherein the output sequence is a sequence of tokens from the first token set representing quantized coordinates of a polygon overlying the object instance in the input image.

15. The method according to claim 14, further comprising: One or more additional output sequences are generated conditioned on the same prompt sequence, and generating a final instance segmentation output includes averaging the dense masks generated from each of the output sequences.

16. A system comprising: one or more computers; as well as One or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations according to any one of claims 1 to 15.

17. One or more computer-readable storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method according to any one of claims 1 to 15.

Images