Performing machine learning tasks by processing images as video

Abstract

A method performed by one or more data processing devices. The method includes: receiving image items; obtaining a mask for selecting portions of the image items; and generating one or more video items from the image items, the one or more video items comprising a corresponding sequence of one or more image frames. Each image frame includes a corresponding portion of the image item selected using the mask. For each image sequence, the mask is incrementally translated over the image items to select corresponding portions of the image items for consecutive image frames in the sequence. The method further includes performing a machine learning task by processing the one or more video items using a machine learning model.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 617,262, filed January 3, 2024. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0003] This manual relates to using machine learning models to process data.

[0004] Machine learning models receive input and generate outputs based on those inputs, such as predicted outputs. Some machine learning models are parametric models and generate outputs based on the received inputs and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models, which use multiple layers to generate outputs from received inputs. For example, deep neural networks are deep machine learning models that include an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification generally describes a method implemented as a computer program on one or more computers in one or more locations, which performs machine learning tasks by processing image items. More specifically, the described technique enables the performance of machine learning tasks by generating one or more video items from image items, and then processing the one or more video items using a machine learning model, such as a multimodal machine learning model, to perform the machine learning task. Processing video items allows the machine learning model to perform machine learning tasks more accurately and / or more efficiently than directly processing image items (i.e., as image items rather than as one or more video items).

[0007] According to a first aspect of this disclosure, a method performed by one or more data processing devices is provided. The method includes: receiving an image item; obtaining a mask for selecting a portion of the image item; and generating one or more video items from the image item, the one or more video items comprising a corresponding sequence of one or more image frames. Each image frame (or a “crop” of the image item) includes a corresponding portion of the image item selected using the mask. For each image sequence, the mask is incrementally translated over the image item to select a corresponding portion of the image item for consecutive image frames in the sequence. The method further includes performing a machine learning task by processing the one or more video items using a machine learning model.

[0008] In some implementations, multiple image frame sequences are generated from a single image item, and these multiple image frame sequences are provided to a machine learning model either as a single video item including these multiple image frame sequences or as multiple video items. Each video item then includes one or more corresponding image frame sequences from the image frame sequence. For example, each video item may include different corresponding image frame sequences generated from the image item.

[0009] In some implementations, the mask can be translated incrementally over image items along a first direction (e.g., parallel to the dimensions of the image item, such as the width or height of the image item) to select corresponding portions of image items for consecutive image frames in a sequence. In some implementations, each frame sequence includes one or more pairs of image frames, each pair including a corresponding first image frame and a corresponding second image frame adjacent to that first image frame in the sequence. The corresponding portions of the image items in the first and second image frames can overlap along the first direction. Such overlap can improve the performance of a machine learning model by allowing it to better consider features of image items spanning more than one image frame. As an example, an image item can be an image comprising text formatted as a table (i.e., tabular data), and image frames can include different corresponding portions of that table (e.g., rows or columns). For example, by overlapping image frames, a machine learning model can be better able to identify that a row or column is part of the same table, rather than part of a different table.

[0010] In some examples, each video item includes at least a first image frame sequence and a second image frame sequence, wherein a corresponding portion of the image item in the second sequence of image frames is offset from a corresponding portion of the image item in the second sequence of image frames along a second direction. The second direction may be perpendicular to the first direction. For example, the first direction may be along the width of the image item, and the second direction may be along the height of the image item, or vice versa. In other words, an image sequence can be obtained by translating a mask in more than one direction; for example, one or more image sequences can be obtained by performing a two-dimensional (2D) scan of the mask on the image item.

[0011] In some examples, an image item may include a three-dimensional image (i.e., an image item comprising 3D pixel or voxel data across three spatial dimensions), in which case a portion of the image item selected using a mask for each image frame may correspond to a three-dimensional region (volume) of the image item rather than a two-dimensional region (area). Each video item may then include a three-dimensional video obtained by translating a mask over the three-dimensional image item (e.g., through the three-dimensional image item). Alternatively, each image frame may be a two-dimensional image representing a different slice of the three-dimensional image or an average or 2D projection of a three-dimensional region. In some examples, a three-dimensional scan of the image item may be performed to generate a sequence of one or more image frames; that is, the mask may be translated along one, two, or three vertical directions within the image item, for example, to perform a 3D scan of the image item. Three-dimensional image items can be obtained using any suitable 3D image generation or capture technique, such as volumetric capture or LiDAR. In some implementations, each image frame may overlap with at least one other image frame along each dimension of the image item.

[0012] In some implementations, the method includes: receiving model input comprising image items for processing by a machine learning model; and then updating the model input by replacing the image items with one or more video items. Performing a machine learning task by processing one or more video items using a machine learning model can then include using the machine learning model to process the updated model input.

[0013] In some implementations, the machine learning model can be a multimodal machine learning model, configured to process a first modal input representing an image item and / or a second modal input representing one or more video items, depending on the model input provided to the machine learning model. Therefore, using a machine learning model to process one or more video items can include using the machine learning model to process model input that includes a second modal input representing one or more video items. In other words, the machine learning model can perform different processing on the model input depending on whether the model input includes image items or one or more video items generated from image items (optionally, also including image items). For example, when processing the second modal input, the machine learning model can consider one or more other image frames in the same sequence as that image frame when processing each image frame, such as one or more other image frames in the sequence before (or after) that image frame. As an example, the machine learning model can apply an attention mechanism that simultaneously focuses on multiple image frames when processing one or more video items.

[0014] In some implementations, the model input provided to the machine learning model may depend on the size or aspect ratio of the image items. For example, if the size of the image items is less than (or in some cases equal to) a predetermined size, the model input may include the image items as a first modal input, while if the size of the image items is greater than (or in some cases equal to) the predetermined size, the model input may include one or more video items as a second modal input.

[0015] In some implementations, the machine learning model is a multimodal machine learning model that processes inputs including a first modal input and a second modal input. The first modal input includes video items, and the second modal input represents modalities other than video. For example, the second modal input could represent one of text items, audio items, or image items. As an example, the second modal input could represent one or more text items, and the machine learning model could be a visual language model configured to generate joint embeddings representing video items and one or more text items, and configured to use these joint embeddings to perform machine learning tasks.

[0016] In some implementations, obtaining the mask involves determining its size based on one or more of the following: the aspect ratio of the image item, the minimum size of the image item, and the maximum size of the image frame that can be processed by the machine learning model. For example, the machine learning model can be configured to receive image items that are less than (or equal to) the maximum size—that is, the maximum width and height (e.g., the number of pixels specified as such as 768×768 pixels, 800×600 pixels, 1024×1024 pixels, etc.). The mask can then, for example, have a width and height equal to the maximum width and height for the machine learning model. In some cases, the mask may alternatively be referred to as a "window." In some implementations, the mask includes binary values ​​for each pixel in the image item, indicating whether the pixel should be included in the corresponding image frame.

[0017] In some other implementations, the method may include resizing each image frame based on the maximum size of the image frames that can be processed by the machine learning model. For example, each image frame may be resized such that the width and / or height of the resized image frame matches the width and / or height of the maximum size of the image frames that can be processed by the machine learning model. Enlarging portions of image items in this way can improve the performance of the machine learning model, for example, by “enlarging” objects or features in the image item that might otherwise receive less attention from the machine learning model.

[0018] In some implementations, the method may include resizing the image item before generating the video item, such that a first dimension of the resized image item is less than or equal to a corresponding first dimension of the mask. As an example, if the mask is incrementally translated along a first direction on the image item to select a corresponding portion of the image item for consecutive image frames in a sequence, the first direction may be along a second dimension of the resized image. For example, the resized image may have a height less than or equal to the height of the mask, and the mask may be translated incrementally along the width of the image item. Alternatively, the resized image may have a width less than or equal to the width of the mask, and the mask may be translated incrementally along the height of the image item.

[0019] In some implementations, resizing image items can preserve their aspect ratio. If the first dimension of the resized image item is smaller than the corresponding first dimension of the mask, the resized image item can be padded with default values ​​or null values ​​to make the first dimension of the resized image item equal to the first dimension of the mask. In some implementations, the image item or mask can be rotated or transposed before resizing, for example, so that the smaller dimension of the image item corresponds to the smaller dimension of the mask. Resizing an image can be referred to as image scaling or image resampling (e.g., downsampling) and can be performed using many common image processing libraries such as OpenCV or scikit-image.

[0020] According to a second aspect of this disclosure, a method for performing a machine learning task using a multimodal machine learning model is provided, executed by one or more data processing devices. The method may include: receiving image items; and providing model input to a multimodal machine learning model configured to process a first modal input representing the image items and / or a second modal input representing one or more video items, depending on the model input, wherein: (i) if the size of the image items is less than (or in some cases, equal to) a predetermined size, the model input includes the image items as the first modal input; and (ii) if the size of the image items exceeds (or in some cases, equal to) the predetermined size, the model input includes one or more video items as the second modal input, the video items being obtained by converting the image items into a corresponding sequence of one or more image frames, each image frame including (e.g., different) corresponding portions of the image items. The method may further include performing a machine learning task by processing the model input using the multimodal machine learning model.

[0021] In other words, a machine learning model can perform different processing depending on whether the model input includes image items and / or one or more video items generated from image items. If the width and / or height of an image item exceeds a corresponding predetermined value (e.g., the maximum width and height of a multimodal machine learning model), the size of the image item can be determined to exceed the predetermined size.

[0022] As an example, converting an image item into one or more image frame sequences may include: obtaining a mask for selecting portions of the image frames; and generating one or more video items from the image items, the one or more video items comprising a corresponding sequence of one or more image frames, each image frame comprising a corresponding portion of the image item selected using the mask, wherein for each image sequence, the mask is incrementally translated over the image item (e.g., along a first direction) to select the corresponding portion of the image item for consecutive image frames in the sequence.

[0023] As an example, converting an image item into a sequence of image frames may include: resizing the image item such that a first dimension of the resized image item is less than or equal to the corresponding first dimension of a mask used to select a portion of the resized image item; and generating a sequence of image frames from the resized image item. Each image frame may include a corresponding portion of the resized image item selected using a mask, wherein the mask is translated incrementally along a second dimension of the resized image item for consecutive frames in the sequence. In some implementations, the image frames may be partially overlapping, which may improve the performance of the machine learning model on the machine learning task in at least some cases. Alternatively, in some implementations, the image frames may not overlap.

[0024] According to a third aspect of this disclosure, a method for performing a machine learning task using a machine learning model is provided, executed by one or more data processing devices. The method includes: receiving an image item (e.g., an image captured by one or more sensors); resizing the image item such that a first dimension (e.g., width or height) of the resized image item is less than or equal to a corresponding first dimension of a mask (e.g., a square or rectangular mask) used to select a portion of the resized image item; generating a video item comprising a sequence of image frames from the resized image item, each image frame including a corresponding portion of the resized image item selected using the mask (e.g., a “crop” of an image). The mask is incrementally translated along a second dimension of the resized image item for consecutive image frames in the sequence. The method further includes performing a machine learning task by processing the video item using a machine learning model.

[0025] In some implementations, the third aspect may additionally include the steps and features described above for the first and second aspects.

[0026] The second dimension of the resized image item can be larger than the second dimension of the mask. A video item can include any appropriate integer number of image frames, such as more than one, more than two, more than five, or more than ten (e.g., 2, 3, 4, 5, 10, 20, 100, etc.). Processing a video item can include providing model input to a machine learning model, which includes image frames and data indicating which image frames belong to the video item (as opposed to the image item).

[0027] The first dimension of a resized image item can correspond to the smaller of the first and second dimensions of the image item. For example, if the width of the image item is greater than its height (e.g., the image item has a "horizontal" orientation), the first dimension can correspond to the height of the image item. Conversely, if the width is less than the height (e.g., the image item has a "vertical" orientation), the first dimension can correspond to the width. The dimensions of the mask can be selected based on the maximum width and height of the image items that the machine learning model is configured to process. For example, the mask can have a width and height equal to the maximum width and height of the machine learning model.

[0028] Resizing image items preserves their aspect ratio. If the first dimension of the resized image item is smaller than the corresponding first dimension of the mask, the resized image item can be padded with default values ​​or null values ​​to make the first dimension of the resized image item equal to the first dimension of the mask. In some implementations, the image item or mask may be rotated or transposed before resizing, for example, so that the smaller dimension of the image item corresponds to the smaller dimension of the mask. Resizing image items can be referred to as image scaling or image resampling (e.g., downsampling) and can be performed using many common image processing libraries such as OpenCV or scikit-image.

[0029] In some implementations, the mask is translated along the second dimension of the resized image item by an increment equal to the length of the mask along the second dimension of the resized image item. A portion of the resized image item for each image frame in the image frame may collectively span the entire resized image item (e.g., the image frame may "tile" the resized image item), or may only span a region of the resized image item. The resized image item may be filled along the second dimension of the image item with default values ​​or null values, for example, such that a predetermined number of image frames can be generated by translating the mask along the second dimension of the resized image item. In some implementations, the image item may be resized such that the second dimension of the resized image is equal to an integer multiple (e.g., 4 times) of the second dimension of the mask.

[0030] In some implementations, the frame sequence comprises one or more pairs of image frames. Each pair of image frames includes a corresponding first image frame and a corresponding second image frame adjacent to the first image frame in the sequence, wherein corresponding portions of the resized images of the first and second image frames overlap along a second dimension of the resized images. For example, such overlap can improve the performance of a machine learning model by allowing it to better consider features of image items that span more than one image frame. As an example, an image item could be an image comprising text formatted as a table (i.e., tabular data), and the image frames could include different corresponding portions of that table (e.g., rows or columns). By overlapping the image frames, the machine learning model can be better able to identify that a row or column is part of the same table, rather than part of a separate table.

[0031] In some implementations, the machine learning model is a multimodal machine learning model that processes inputs including a first modal input and a second modal input. The first modal input includes video items, and the second modal input represents text, audio, or video items. The second modal input may represent one or more text items, and the machine learning model is a visual language model configured to generate joint embeddings representing video items and one or more text items, and configured to use the joint embeddings to perform machine learning tasks.

[0032] In some other implementations, the machine learning model is a multimodal machine learning model, configured to perform a machine learning task by processing (input) image items if the size of the image item is less than or equal to a predetermined size. In such a case, video items may not be generated, but if one or more video items are generated, the image items can be processed by the multimodal machine learning model along with the video items, which can improve the multimodal machine learning model's performance on the machine learning task.

[0033] For example, the machine learning model can be a multimodal machine learning model configured to process a first modal input representing an image item and / or a second modal input representing a video item, depending on the model input provided to the machine learning model. If the size of the image item is less than (or in some cases, equal to) a predetermined size, the model input can include the image item as the first modal input. If the size of the image item is greater than (or in some cases, equal to) the predetermined size, the model input can include the video item as the second modal input. In other words, the machine learning model can perform different processing depending on whether the model input includes an image item and / or a video item generated from the image item. In some implementations, the size of the image item can be determined to be less than a predetermined size if the width and / or height of the image item is less than a predetermined value.

[0034] For any of the first, second, and third aspects, the machine learning task may include one or more of the following: object or action detection task, classification task, caption generation task, question answering task, natural language translation task, character or word recognition task, image or audio generation task, or computer language generation task. Image items may include images of the document, and the output of the machine learning task may depend on the text and / or one or more images (e.g., graphics or charts) in the document. For example, a document may be one or more of the following: webpage, infographic, form (e.g., an order form with details of items and / or services to be purchased), map (e.g., a map of a geographical area or building), receipt (e.g., a receipt listing purchased items), photographic film (e.g., a roll or reel of photographic film), and design drawings of objects or buildings (e.g., blueprints or architectural drawings, circuit diagrams). The document may also include screenshots of images displayed on a computer system's monitor.

[0035] In some implementations, an image item is an image of a scene comprising multiple objects arranged along a direction corresponding to a second dimension of the image item, and the machine learning task involves identifying one or more objects among the objects (and optionally counting one or more objects). For example, the objects could be items arranged on shelves in a supermarket, warehouse, or factory. The model output could then include, for example, an inventory of the objects. Treating the image item as a video item allows the machine learning model to more accurately identify one or more objects and / or count one or more objects.

[0036] In some implementations, the machine learning task includes an agent control task, where an agent interacts with an environment to perform the agent control task, where the image items include observations of the environment, and where the output of the machine learning model is used to select one or more actions to be performed by the agent in response to the observation in the environment. The environment can be a real-world environment or a simulated environment. For example, the machine learning model can be trained in a simulated environment and then deployed in a real-world environment.

[0037] According to a further aspect of this disclosure, one or more non-transitory computer storage media are provided that store instructions which, when executed by one or more computers, cause one or more computers to perform the operation of the corresponding method of the first aspect or the second aspect.

[0038] According to a further aspect of this disclosure, a system is provided comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the corresponding method of the first aspect or the second aspect.

[0039] The system may additionally include an image sensor (e.g., a camera or scanner) configured to acquire image items and provide them to one or more computers for performing machine learning tasks.

[0040] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0041] Using machine learning models to process video items can include processing image frames consisting of two or more image frames sequentially, individually, or in batches. Since each image frame is typically smaller than the received image item, the (instantaneous) amount of memory resources required to process a video item can be less than when the image item is processed by the machine learning model as a (single) image item (relative to the video item). In some cases, some image frames can be processed in parallel (e.g., multiple batches of image frames).

[0042] Generally, machine learning models can be configured to receive image items smaller than their maximum size—that is, the maximum width and height (specified as the number of pixels, such as 768×768 pixels, 800×600 pixels, 1024×1024 pixels, etc.). In some existing image processing methods, image items are therefore resized so that both the width and height of the resized image are smaller than the maximum width and height, in order to allow them to be processed by the machine learning model. However, such resizing (resampling) can reduce the amount of information available to the machine learning model, resulting in relatively poor performance of the machine learning task, especially for image items with very different widths and heights—that is, low or very high aspect ratios (or whose maximum width and height are very different from those of the machine learning model). As described in this paper, the amount of information lost due to resizing image items can be reduced. For example, for image items whose width is much greater than their height (e.g., landscape or panoramic images), the image item can be resized such that the height of the resized image item is less than or equal to the maximum height that can be processed by a machine learning model, while the width of the resized image item remains greater than the maximum width that can be processed by a machine learning model. In this case, each image frame in the video item can have a width less than or equal to the maximum width that can be processed by a machine learning model, where the image frames collectively span the width of the resized image item. Therefore, more information in the image item can be preserved while still allowing the machine learning model to process the resized image item.

[0043] The improved performance of the machine learning model can also be achieved by using a mask size smaller than the maximum size of an image item that the machine learning model can process, and then resizing each image frame in the image frame to the maximum size of an image item that the machine learning model can process. Due to the increased size of the image frames, the machine learning model can, for example, be better able to focus on the features of the image frames.

[0044] Generally, a machine learning model can be a neural network model with multiple learnable parameters (such as weights) that are updated during training. Such a neural network model can have any suitable architecture, such as attention-based neural network architectures (e.g., Transformer architecture), convolutional architectures, fully connected architectures, or any other suitable neural network architecture. The model can include any suitable number (e.g., 10 layers, 100 layers, or 1000 layers) and any suitable type of neural network layers (e.g., convolutional layers, attention layers, fully connected layers, recurrent layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers or as a directed graph of layers). Some implementations of the model described in this paper can be described as “large” multimodal models, e.g., having more than 10... 10There are learnable parameters, but the application of the described technique is not limited to such a model.

[0045] Generally, a multimodal machine learning model has multiple multimodal inputs (and optionally, one or more further multimodal inputs) configured to receive a first multimodal input and a second multimodal input. As used herein, “modality” refers to the type of data, and therefore a multimodal machine learning model is a model capable of handling multiple different types of data. In some implementations, there may be more than two distinct multimodal inputs, each configured to receive a different type of data.

[0046] For example, a first multimodal input can be configured to receive an image item, which may include image data defining the color or intensity values ​​of pixels in an image. As used herein, an "image" (image item) includes a point cloud (e.g., a LiDAR point cloud), and the image data may additionally or alternatively define the locations of points in the point cloud. A second multimodal input can be configured to receive a video item. A video item may include image data defining the color or intensity values ​​of pixels in each image frame in a sequence of image frames (i.e., two or more image frames in a specified order). Video items can be generated from image items, for example, such that each image frame in the sequence includes a corresponding portion of the image item. The model input may optionally include one or more further multimodal inputs, such as text input. Image items may include individual images, portions of images (e.g., patches), frames of video items, etc.

[0047] As another example, the first multimodal input can be configured to receive image items, and the second multimodal input can include text input configured to receive a sequence of text. The text can be text in one or more natural languages, or text in a computer language, or both. The computer language can be any formal language used for communicating with a computer, such as a markup language, or a command or configuration language, or a data exchange language (such as JSON), or a programming language. The text can be received, for example, as a series of encoded characters (e.g., UTF-8 encoded characters); such "characters" can include Chinese and other similar characters, as well as logograms, syllabograms, and the like. The text can include instructions and / or contextual data for performing machine learning tasks. In some cases, the text can include transcribed speech.

[0048] Multimodal machine learning models may include one or more multimodal input encoders configured to process one or more multimodal inputs from a multimodal input, for example, using one or more convolutional layers, attention layers, fully connected layers, or recurrent layers, to generate an encoded version of the multimodal input. Generally, such encoders can implement any form of encoding suitable for the data type to be encoded. In the case where the multimodal input includes images or videos, this can be encoded as, for example, a sequence of visual words selected from a vocabulary of visual words, or as representations of different objects in the visual input. In the case where the multimodal input includes text input, the machine learning model may include a text encoder that processes the text sequence to represent the text as a series of text words from a vocabulary, for example, each representing a word, word fragment, or character in natural or computer language. Visual words may, but need not, interleave with the text words processed by the model.

[0049] This multimodal machine learning model can be configured to jointly process an encoded version of a text sequence and an encoded version of another multimodal input (e.g., an image or video) to generate a model output that defines the outcome of a machine learning task. The multimodal machine learning model can be a visual language model.

[0050] Model output can include any form of output suitable for the machine learning task performed by the machine learning model. For example, model output can include text in natural or computer language that defines the outcome of the task, such as for tasks like image caption generation, visual question answering, or object detection or instance segmentation. Additionally or alternatively, model output can include data defining image, video, or audio objects, for example, in generative tasks; or model output can include non-textual action selection data for selecting actions to be performed by an agent controlled by the model. As another example, model output can also or alternatively define intermediate steps to be performed during the task, such as calls to a software API for a software tool used in performing the task; the multimodal input can then receive output from that software tool, which is used to generate the final model output for performing the task. In some cases, model output can be multimodal.

[0051] Such multimodal machine learning models can be trained using very large (but potentially noisy) datasets, where text is paired with images and / or with one or more other types of data, such as audio data or data related to the actions of an agent acting in an environment to perform various tasks. Such models can be trained, for example, using self-supervised learning. The pairings may often be imperfect, and the training dataset may or may not include any real-world examples of the specific task to be performed; however, the ability to perform a specific task may still emerge. Numerous examples of suitable, publicly available training datasets exist.

[0052] Some example multimodal machine learning models that can be used with the techniques described in this paper include: Flamingo (Alayrac et al., arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al., arXiv:2209.06794); PaLI-X (Chen et al., arXiv:2305.18565); and MIRASOL3B (Piergiovanni et al., arXiv 2311.05698).

[0053] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims.

[0054] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of a preprocessor system.

[0056] Figure 2A This is a view of example image items showing the location of the mask used to obtain the image frame sequence.

[0057] Figure 2B Is using Figure 2A The position of the mask in the image is a schematic diagram of the image frame sequence obtained from the image item in Figure 2.

[0058] Figure 3A It indicates the location of another mask. Figure 2A A view of the image items.

[0059] Figure 3B Is using Figure 3A The position of the mask is a schematic diagram of the first image frame sequence obtained from the image items in Figure 2.

[0060] Figure 3C Is using Figure 3A The position of the mask is a schematic diagram of the second image frame sequence obtained from the image items in Figure 2.

[0061] Figure 4A and Figure 4B It is a flowchart of the process used to perform machine learning tasks.

[0062] Figure 5A and Figure 5B It is a flowchart of another process used to perform machine learning tasks.

[0063] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0064] The systems and methods described in this specification can improve the ability of machine learning models to perform machine learning tasks by converting image items into one or more video items to replace or supplement image items for processing by machine learning models.

[0065] Figure 1 An example of a preprocessor system 100 is shown, which is a system implemented as a computer program on one or more computers (data processing devices) in one or more locations, wherein the systems, components and techniques described below are implemented.

[0066] The preprocessor system 100 is configured to receive a model input 102 including an image item 104, and to process the model input 102 to generate an updated model input 106 including a video item 108 (or more than one video item). The video item 108 may, for example, replace the image item 104 in the updated model input 106, or may provide the video item in addition to the image item 104 in the updated model input 106. As in this example, the model input 102 and the updated model input 106 may additionally include one or more input items besides the image item 104, such as a text item 110.

[0067] The updated model input 106 can be processed by machine learning model 112 to perform a machine learning task that generates a corresponding model output 114 based on the updated model input 106. In general, the machine learning task can be any machine learning task that can be performed using model input 102 or the updated model input 106. As an example, the machine learning visual language model 112 can be a visual language model configured to generate a joint embedding representing video item 108 (or image item 104) and text item 110, and configured to use this joint embedding to perform a machine learning task, such as object recognition or image classification.

[0068] The preprocessor system 100 includes an image sequence generator 116 configured to process image items 104 to generate one or more sequences 118 of image frames 120 using a mask 122. Each image frame 120 includes a corresponding portion of the image item 104 selected using the mask 122. The image sequence generator 116 is configured to incrementally (stepwise) translate the mask over the image items 122 to select corresponding portions of the image items 104 for consecutive image frames 120 in each sequence 118. For example, the image sequence generator 116 may be configured to translate the mask 122 over the image items 104 along a first direction to select corresponding portions of the image items 104 for consecutive image frames 120 in each sequence 118. Each image frame includes a different corresponding cropped piece (i.e., a different corresponding region of the image item) of the image item obtained using the mask.

[0069] In some implementations, mask 122 can be a rectangle (e.g., a square) with fixed dimensions (e.g., width and height), such that the same mask 122 is used for different image items 104. In such cases, image item 104 can be resized, for example, such that the first dimension (e.g., width or height) of the resized image item is less than or equal to the corresponding first dimension of mask 122. That is, image item can be resized such that the mask spans or exceeds the width or height of the image item.

[0070] Alternatively, the dimensions of mask 122 can be determined by preprocessor system 100 based on image item 104 (e.g., based on the dimensions of image item 104). That is, the dimensions of mask 122 can be adapted to image item 104. As an example, the dimensions of mask 122 can be determined based on the minimum values ​​of the width and height of image item 104. For example, the width and / or height of the mask can be equal to the minimum values ​​of the width and height of image item 104 divided by a scaling factor greater than one. In some examples, the dimensions of mask 122 may additionally be required to have at least a minimum value (i.e., a minimum width and / or height).

[0071] In some implementations, the preprocessor system 100 may be configured to generate one or more video items 108 only if the image item 104 meets one or more predetermined criteria (e.g., criteria based on one or both of the dimension of the image item and / or the aspect ratio of the image item 104). For example, the preprocessor system 100 may be configured to generate one or more video items 108 if the width or height of the image item 104 exceeds a predetermined first threshold and / or the aspect ratio of the image item 104 exceeds a predetermined second threshold.

[0072] If image item 104 does not meet one or more criteria, the preprocessor system 100 may provide model input 102 (instead of the updated model input 106) to the machine learning model 112 for the execution of machine learning tasks.

[0073] In some implementations, image frame 120 can be resized, for example, to have the same dimensions as the maximum or minimum image size that can be processed by machine learning model 112. Any image resizing (scaling) technique can be used to resize image item 104 or image frame 120, such as nearest neighbor interpolation, bilinear interpolation, spline interpolation, Fourier transform-based scaling methods, neural network scaling methods, and so on.

[0074] Figure 2A An exemplary image item 200 is shown, which in this case depicts a scene of a physical environment. For this example, the machine learning task could be detecting one or more objects in the scene, classifying the scene, or determining a series of actions by a robotic agent to perform an agent control task (e.g., a manipulation task or a navigation task) in the physical environment, and so on. Image item 200 is divided into a first image portion (region) 202A and a second image portion (region) 202B obtained by translating a mask 204 from left to right (i.e., across the width of image item 200) on image item 200. In this example, the first image portion and the second image portion 202A overlap such that each of the consecutive (i.e., first and second) image portions 202A, 202B includes a common region 202C of image item 200. In this example, the mask 204 spans the entire height of image item 200, such that image item 200 can be covered using a single row of image portions 202A, 202B.

[0075] Figure 2BImage sequence 208 is shown, comprising a first image frame 206A and a second image frame 206B. The first image frame includes a first image portion 202A, and the second image frame includes a second image portion 202B. Image sequence 208 can be processed as a video item by machine learning model 112 of FIG. 2. Including a common region 202C of image item 200 in each of the first image frame 206A and the second image frame 206B can improve the ability of machine learning model 112 to infer the scene depicted in image item 200 using image sequence 208.

[0076] Figure 3A It shows Figure 2A Image item 200, image portions 302A-C of the first row 300A of image item 200, and image portions 302D-F of the second row 300B of image item 200. Image portions 302A-D in each of rows 300A and 300B are obtained by translating another mask 304 across image item 100. In this example, mask 304 does not span the entire height of image item 200, such that image portions 302A-F of two rows 300A and 300B are required to cover image item 200, wherein image portions 302D-F of the second row 300B are offset vertically from image portions 302A-C of the first row 300A. As in Figure 2A In the example, consecutive image portions 302A-C and 302D-F in each of the first row 300A and the second row 300B overlap horizontally. Furthermore, in this example, image portion 302D-F of the second row 300B overlaps vertically with image portion 302A-C of the first row 300A.

[0077] Figure 3B A first image sequence 308A is shown, comprising multiple image frames 306A-C, each image frame including Figure 3A The corresponding image portions in the first row of the image portion 300A and the image portion 302A-C.

[0078] Figure 3C Similarly, a second image sequence 308B comprising multiple image frames 306D-F is shown, each image frame including Figure 3AThe corresponding image portions in the image portions 302D-F of the second row 300B of the image portion. Image frames 306D-F of the second image sequence 308B are obtained by translating the mask 304 across the image item 200 in the same direction as the image frames 306A-C used to obtain the first image sequence 308A. However, image frames 306D-F of the second image sequence 308B can alternatively be obtained by translating the mask 304 in the opposite direction to the direction used to obtain the image frames 306A-C used to obtain the first image sequence 308A. As another example, image frames of an image sequence can be obtained by translating the mask over the image item in more than one direction (e.g., along a path with components in two vertical directions). For example, the path can be one of the following: row-first or column-first raster scan, outward or inward spiral path, serpentine or meandering path—e.g., a continuous path including segments (e.g., rows of columns) in opposite directions, etc. In some examples, an image sequence can be generated by translating a mask along a path that covers the entire image item, such that the image sequence includes image frames that collectively cover the entire image item. Therefore, translating a mask on an image item can include performing a 1D or 2D “scan” of the image item.

[0079] The first image sequence 308A and the second image sequence 308B can be provided to the machine learning model 112 in a single video item 108 or in the corresponding first video item and second video item.

[0080] It should be understood that more image segments, and optionally more rows of image segments, can be obtained by using smaller masks or by increasing the amount of overlap between image segments in each row or between image segments in consecutive rows. For example, each image sequence may include more than two image frames, more than three image frames, more than five image frames, and so on. In some examples, one or more (e.g., all) image sequences in an image sequence may be provided in a single video item.

[0081] Although Figure 2A and Figure 2B as well as Figure 3A and Figure 3BProcessing a horizontal image item 200 has been illustrated; however, it should be understood that a vertical image item can be processed in a similar manner, for example, by translating a mask vertically over the image item to generate one or more columns of image portions, from which a corresponding image sequence can be obtained. Similarly, for practical reasons, the exemplary image items shown in the figures are 2D images; however, it should be understood that the systems and methods described in this specification can also be applied to 3D image items, such as image items characterizing a 3D volume of a physical environment captured using one or more sensors (e.g., a LiDAR camera or camera array). For example, a mask can define a 3D region that is translated across a 3D volume to obtain one or more sequences of 3D image frames; that is, one or more sequences of image frames can be obtained by “scanning” the mask across the 3D volume. For example, a machine learning model can process 3D video items to better infer about the physical environment.

[0082] Figure 4A A process 400 that can be executed by one or more data processing devices (computers) is shown. For example, the process can be performed by, for example, Figure 1 The preprocessor system and machine learning model shown are used to achieve this.

[0083] Process 400 includes receiving image items (step 402) and obtaining a mask for selecting portions of the image items (step 404). The system then generates one or more video items from the image items (step 406). Process 400 further includes performing a machine learning task by processing the one or more video items using a machine learning model. An example of a machine learning task is described below.

[0084] Figure 4B Step 406, which can be performed to generate one or more video items from an image item, is illustrated. A mask is applied to the image item (step 410), for example, by selecting an initial position of the mask relative to the image item. Image frames are then generated by cropping the image item using the mask (step 412). It is then determined (step 414) whether the mask has reached the edge of the image item. If the mask has not yet reached the edge of the image item, the mask is translated along a first direction (e.g., horizontally or vertically) (step 418), and steps 410-414 are repeated. Once the mask has reached the edge of the image item, a video item is generated from the sequence of image frames obtained by repeatedly performing steps 410-414.

[0085] Figure 5A Another process 500, which can be executed by one or more data processing devices, is shown. For example, this process can be performed via, as... Figure 1 The preprocessor system and machine learning model shown are used to achieve this.

[0086] Process 500 includes: receiving an image item (step 502); providing the image item-based model input to a multimodal machine learning model (step 504); and performing a machine learning task by processing the model input using the multimodal machine learning model.

[0087] Figure 5B A sub-process of step 504, which provides image-item-based model input to a multimodal machine learning model, is illustrated. This sub-process includes determining whether the size of the image item is less than or equal to a predetermined size (step 510). If the image item's size is less than the predetermined size, the model input includes the image item as a first modal input (step 512). For example, if the image item's width is less than a predetermined width and its height is greater than a predetermined height, it can be determined that the image item's size is less than the predetermined size. If the image item's size exceeds the predetermined size, step 514 is performed to obtain one or more video items by converting the image item into a corresponding sequence of one or more image frames. Each image frame includes a corresponding portion of the image item. Then, one or more video items are included in the model input as a second modal input (step 516).

[0088] Generally speaking, machine learning tasks performed by machine learning models can include one or more of the following examples of machine learning tasks.

[0089] For some tasks, the multimodal input represents an image or video generated from an image, as previously described. Images can be obtained from a camera or other imaging devices that capture images from the real-world environment. Generally, the tasks described below can be those that require spatial awareness or other context from the image item. For example, a cue word might ask, “What is the object in the top left corner?”

[0090] As an example, a task could include an object or action detection task. Task-specific training data items could include image or video items containing a sequence of text that includes one or more objects or actions (e.g., an image of a scene performing one or more actions) and the objects or actions. The sequence of text could describe or otherwise annotate the objects or actions and could include text that gives the bounding box coordinates of the objects or actions. After training, when the model is used for inference, the model output could include or represent text describing or otherwise annotating the detected objects or actions in the image, and could include the bounding box coordinates of the detected objects or actions, such as "10 20 90 100 cat 20 30 100 100 dog".

[0091] As another example, the task can include a classification task, such as an object or action classification task. Task-specific training data items can include image or video items containing sequences of one or more objects or actions along with text. The text sequence can describe the objects or actions or otherwise classify them. After training, when the model is used for inference, the model output can include data that classifies the objects or actions in the image items into one of several classes, such as text.

[0092] As another example, the task can include an image item description task, such as a caption generation task (as used herein, which includes audio description tasks for explaining what is shown in an image). Task-specific training data items can include image items (and / or video items generated from image items) and sequences of text describing those image items. After training, when the model is used for inference, the model output can include data describing the image items, such as text. For example, the model output could provide captions or descriptions of the image items, or it could count objects in the image items, or it could provide some other form of description of the image items.

[0093] As another example, the task could include an image question-answering task. Task-specific training data items could include image items (and / or video items generated from image items) and sequences of text describing those image items. After training, when the model is used for inference, the model output could include data (e.g., text) that answers questions about the image items specified in the sequence of prompts in the text, as described above. This could be used, for example, to answer questions about visual drawings and charts.

[0094] As another example, the task can include character or word recognition tasks, such as OCR (Optical Character Recognition) tasks. Task-specific training data items can include image items (and / or video items generated from image items) and sequences of text including the text depicted in the image. After training, when the model is used for inference, the model output can include text representing, for example, characters or words in the input image item presented in natural language.

[0095] As another example, the task can include a static or moving image generation task. Task-specific training data items can include image items (and / or video items generated from image items) and a sequence of text describing the image item. After training, when the model is used for inference, the model output can include data that may include image items or video items, such as image data defining pixel values ​​for static or moving images, and a sequence of text in the model's multimodal inputs that can describe or characterize the image item or video item to be generated.

[0096] As another example, the task may include a computer language text generation task. Task-specific training data items may include image items or video items and sequences of text in computer language used to generate the image items, video items, or audio items. After training, when the model is used for inference, the model output may include text in that computer language or another computer language used to generate or render the image items, video items, or audio items (e.g., web pages, graphs, or charts).

[0097] In another example of a computer language text generation task, task-specific training data items may include image items (and / or video items generated from image items) and a sequence of text in computer language for performing tasks related to the image items, such as analyzing the content of the image items to provide analysis results or providing a search to search for information related to the content of the image or video items. The computer language in the model output may include computer language used to invoke functions or call one or more external APIs. As just one example, such output may be formatted as a JSON object. As previously described, the sequence of text in the multimodal input can define the tasks to be performed in relation to the input image items (and / or videos generated from the input image items). For example, the task may involve manipulating a specific type of data that could benefit from accessing an API, such as mathematical data, date / time related data, scientific data, recent data that may be later than the model's training (accessible via search functions or APIs), and so on. After training, when the model is used for inference, the model output may include text in that computer language or another computer language for performing tasks related to the image or video items (e.g., as described above). The method may then include performing the task using text rendered in the computer's language.

[0098] Generally, when the model output includes text, this can be provided as speech representing the text.

[0099] In some implementations, the machine learning task includes an agent control task in which the agent interacts with the environment to perform an agent control task. In these implementations, the model input includes image items, which include observations representing the environment captured, for example, from the real-world environment by a camera or other imaging device. The model input may additionally include a sequence of text defining the task to be performed by the agent.

[0100] Task-specific training data items may include sequences of text representing one or more actions of an agent, and image items representing observations of the environment (and optionally, other or similar modal data representing observations of the environment). After training, when the machine learning model is used for inference, the model output includes action selection output, such as text, for selecting one or more actions the agent will perform in response to the observed actions in the environment. For illustration, the model output can define actions as text, such as "A: 132 114 128 5 25 156", which can be translated into control signals for a mechanical agent (such as a robot), such as "...". As another example, the action selection output can also, or alternatively, define one or more low-level skills, such as a vocabulary of previously learned skills. As previously mentioned, a sequence of text in the model's multimodal input can describe the task to be performed, such as "What action should the robot take to [perform task]".

[0101] In some implementations of agent control, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating or navigating within it. Actions are actions taken by the mechanical agent in the real-world environment to perform tasks. For example, the agent may be a robot or other mechanical agent that interacts with the environment to complete a specific task (e.g., locating or manipulating an object of interest in the environment, moving the object of interest to a designated location in the environment, or navigating to a designated destination in the environment). In these implementations, observation may include one or more of the following: images, object position data, and sensor data used to capture observations as the agent interacts with the environment. Actions may define control signals used to control the robot or other mechanical agent, such as position, torque, or other control signals for parts of the mechanical agent, or higher-level control commands.

[0102] In some implementations of agent control, the agent can be a software agent, i.e., a computer program, configured to perform tasks. The following are some examples where the agent is a software agent.

[0103] As an example, the environment could be integrated circuit design, and the task could be a routing task for wiring interconnects of the integrated circuit. Observations could include component locations and / or interconnects, and actions could include component placement or interconnect wiring actions. An integrated circuit with interconnects wired as determined could then be manufactured.

[0104] As another example, the environment can be a real-world computing environment, and the task can be managing the distribution of jobs or tasks across computing resources, such as on mobile devices and / or in data centers. Observations can include observations of computing resources, such as computing or storage capacity or internet-accessible resources, or observations related to operations on computing resources in processing jobs or tasks; and actions can include assigning jobs or tasks to specific computing resources.

[0105] As another example, the environment can be a real-world computing environment, and the task is, for example, managing the processing of a queue of consecutively arriving jobs via one or more real-world servers. Observations can include observations of the departure times of consecutive jobs, or the time intervals between the departures of consecutive jobs, or the time spent by the server processing each job, or the arrival times of consecutive jobs, or the time intervals between the arrivals of consecutive jobs, or observations of data characterizing the types of jobs. Actions can include the action of assigning a specific job to a specific computing resource.

[0106] As another example, the environment may include a real-world computer system or network, and the task may be to maintain the security of the computer system or network. Observations may include any observations characterizing the operation of the computer system or network, and actions may include actions to control operations, such as limiting or correcting abnormal or undesirable operations caused, for example, due to the presence of a virus or other security vulnerability.

[0107] As another example, the environment may include a data packet communication network environment, and the task may be routing packets of data through the communication network. Actions may include data packet routing actions, and observations may include, for example, observations of a routing table that includes routing metrics such as route path length, bandwidth, load, hop count, path cost, latency, maximum transmission unit (MTU), and reliability metrics.

[0108] In some implementations of intelligent agent control, the intelligent agent can be a human intelligent agent, and the environment can be a real-world environment. For example, the intelligent agent can be a human user of a digital assistant (such as a smart speaker, smart display, or some other device used to instruct the user to perform actions). The task can be any real-world task the user wishes to perform. Observations can be obtained from an observation capture subsystem (e.g., a monitoring system such as a video camera or sound capture system) used to capture visual and / or audio observations of the user performing the task. Actions can include instructions in the form of, for example, text, images, video, or audio data (such as speech) guiding the user to perform the task.

[0109] In this specification, the term "configured" is used in connection with computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or combinations thereof that enable it to perform specific operations or actions during operation. For example, configuring a system may involve installing a software library with specific algorithms, updating firmware with new instructions for data manipulation, or adding hardware components to gain enhanced processing power. Similarly, one or more computer programs are "configured" to perform those intended operations or actions when they contain instructions that, when executed by a computing device or hardware, cause the device to perform specific operations or actions.

[0110] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuit systems, software, firmware, computer hardware (covering the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be implemented as one or more computer programs, which are essentially modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a computing device or hardware or for controlling the operation of a computing device or hardware. The storage medium can be a storage device such as a hard disk drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions can be encoded on transmitted signals such as machine-generated electrical, optical, or electromagnetic signals, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations can leverage emerging technologies such as quantum computing or neuromorphic computing for specific applications and can be deployed in distributed or cloud-based environments where components reside on different machines or within cloud infrastructure.

[0111] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, computing devices or hardware may also include code that creates the execution environment for computer programs. This code can take the form of processor firmware, protocol stacks, database management systems, operating systems, or combinations of these elements. In the context of general-purpose computing on a graphics processing unit (GPGPU), in which code specifically designed for GPU execution (often referred to as kernels or shaders) is employed, embodiments can particularly benefit from leveraging the parallel processing capabilities of GPUs. Similarly, TPUs excel at running optimized tensor operations crucial to many machine learning algorithms. By utilizing these accelerators and their specialized programming models, the system can achieve significant acceleration and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in fields such as computer vision, natural language processing, and robotics.

[0112] Computer programs (also known as software, applications, modules, scripts, code, or simply programs) can be written in any programming language, including compiled or interpreted languages, as well as declarative or procedural languages. They can be deployed in various forms, such as standalone programs, modules, components, subroutines, or any other unit suitable for use in a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., a script within a markup language document), residing in a dedicated file, or distributed across multiple coordination files (e.g., files storing modules, subroutines, or code segments). Computer programs can execute on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected via data communication networks. The specific implementation of a computer program can involve a combination of traditional programming languages ​​and programming for GPGPUs or TPUs utilizing specialized languages ​​or libraries designed for them, depending on the hardware platform chosen and the desired performance characteristics.

[0113] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components mounted on one or more computers, which may be located at a single site or distributed across multiple locations. In some cases, one or more dedicated computers may be used for a particular engine, while in others, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning may include data preprocessing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of an engine will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0114] The processes and logic flows described in this specification can be executed by one or more programmable computers, which run one or more computer programs to perform functions by manipulating input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be used to implement aspects of these processes and logic flows concurrently, thereby significantly accelerating execution. This approach offers significant advantages for computationally intensive tasks common in AI and machine learning applications, such as matrix multiplication, convolution, and other operations exhibiting high parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedup and efficiency gains can be achieved compared to relying solely on CPUs. Alternatively, or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), to achieve even higher performance or energy efficiency in specific use cases.

[0115] Computers capable of executing computer programs can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators typically work in conjunction with the CPU, handling specialized computations, while the CPU manages overall system operation and other tasks. Typically, the CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The basic components of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of the processing unit and memory will depend on factors such as the complexity of the AI ​​model, the amount of data being processed, and the desired performance and latency requirements. Implementations can be carried out on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. Systems may include storage devices such as hard disks, SSDs, or flash memory for persistent data storage.

[0116] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and storage devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or durability.

[0117] To facilitate user interaction, embodiments of the subjects described in this specification can be implemented on computing devices equipped with display devices (such as liquid crystal displays (LCDs) or organic light-emitting diode (OLED) displays) for presenting information to users. Input can be provided by the user through various means, including keyboards, touchscreens, voice commands, gesture recognition, or other input modalities, depending on the specific device and application. Additional input methods may include sound, voice, or tactile input, while feedback to the user may take the form of visual, auditory, or tactile feedback. Furthermore, the computer can interact with the user by exchanging documents with the user's device or application. This may involve sending web content or data in response to a request, or sending and receiving text messages or other forms of messages via mobile devices or messaging platforms. The choice of input and output modalities will depend on the specific application and the desired form of user interaction.

[0118] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX. These frameworks provide comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0119] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These components may include: backend components, such as backend servers or cloud-based infrastructure; optional middleware components, such as middleware servers or application programming interfaces (APIs), to facilitate communication and data exchange; and frontend components, such as client devices having a user interface through which users can interact with the implemented subject matter, a web browser, or an app. For example, the described functionality may be implemented only on the client device (e.g., for on-device machine learning) or deployed as a combination of frontend and backend components for more complex applications. Where present, these components may be interconnected using any form or medium of digital data communication, such as communication networks, such as local area networks (LANs) or wide area networks (WANs), including the Internet. The specific system architecture and component selection will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0120] Computing systems can include geographically separated clients and servers that interact via communication networks. The specific type of network (such as a local area network (LAN), wide area network (WAN), or the Internet) will depend on the accessibility and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. Depending on the nature of the data being exchanged and the system's security requirements, these protocols may include HTTP, TCP / IP, or other specialized protocols. In some embodiments, the server transmits data or instructions to a user device acting as a client, such as a computer, smartphone, or tablet. The client device can then process the received information, display the results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interaction between the user and the system, enabling a wide range of applications and functionalities.

[0121] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope of any claims, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0122] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to achieve the desired result. In some contexts, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0123] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: Receive image items; Obtain a mask for selecting a portion of the image item; One or more video items are generated from the image items, the one or more video items comprising a corresponding sequence of one or more image frames, each image frame comprising a portion of the image item selected using the mask, wherein, for each image sequence, the mask is incrementally translated over the image item to select a portion of the image item for consecutive image frames in the sequence; as well as A machine learning task is performed by processing one or more video items using a machine learning model.

2. The method as described in claim 1, wherein, The mask is translated incrementally over the image item along a first direction to select a corresponding portion of the image item for consecutive image frames in the sequence.

3. The method as described in claim 2, wherein, Each image frame sequence includes one or more pairs of image frames, each pair of image frames including a corresponding first image frame and a corresponding second image frame adjacent to the first image frame in the sequence, wherein the corresponding portions of the image items of the first image frame and the second image frame overlap along the first direction.

4. The method as described in claim 2 or 3, wherein, Each video item includes at least a first image frame sequence and a second image frame sequence, wherein a corresponding portion of the image item in the second sequence of image frames is offset from a corresponding portion of the image item in the second sequence of image frames along a second direction.

5. The method as described in any of the preceding claims, wherein, The machine learning model is configured to process a first modal input representing an image item and / or a second modal input representing one or more video items, depending on the model input, and wherein processing the one or more video items using the machine learning model includes: The machine learning model is used to process model inputs that include second modal inputs representing the one or more video items.

6. The method as described in any of the preceding claims, wherein, The machine learning model is a multimodal machine learning model, and the method further includes: providing model input to the machine learning model, the model input including a first modal input for modalities other than videos and a second modal input including the one or more video items.

7. The method of claim 6, wherein, The first modal input represents one or more text items, and the machine learning model is a visual language model configured to generate joint embeddings representing the video items and the one or more text items, and configured to use the joint embeddings to perform the machine learning task.

8. The method as described in any of the preceding claims, wherein, Obtaining the mask includes determining the size of the mask based on one or more of the following: the aspect ratio of the image item, the minimum dimension of the image item, and the maximum size of the image frame that can be processed by the machine learning model.

9. The method of any of the preceding claims, further comprising resizing each image frame according to the maximum size of the image frame that can be processed by the machine learning model.

10. The method of any of the preceding claims, further comprising, prior to generating the video item, resizing the image item such that a first dimension of the resized image item is less than or equal to a corresponding first dimension of the mask.

11. The method of claim 10, wherein, The mask is incrementally translated over the image item along a first direction to select a corresponding portion of the image item for consecutive image frames in the sequence, and the first direction is along the second dimension of the resized image.

12. The method according to any one of claims 10 to 11, wherein, The image item is resized while preserving its aspect ratio.

13. A method executed by one or more data processing devices for performing a machine learning task using a multimodal machine learning model, the method comprising: Receive image items; A model input is provided to a multimodal machine learning model, the multimodal machine learning model being configured to process a first modal input representing an image item and / or a second modal input representing one or more video items, depending on the model input, wherein: (i) If the size of the image item is smaller than a predetermined size, then the model input includes the image item as a first modal input; and (ii) If the size of the image item exceeds the predetermined size, the model input includes one or more video items as a second modal input, the one or more video items being obtained by converting the image item into a corresponding sequence of one or more image frames, each image frame including a corresponding portion of the image item; as well as Machine learning tasks are performed by processing the model input using the multimodal machine learning model.

14. The method of claim 13, wherein, If the width and / or height of the image item exceeds a predetermined value, then the size of the image item exceeds the predetermined size.

15. The method of claim 13 or 14, wherein, Converting the image item into an image frame sequence includes: Obtain a mask for selecting a portion of the image item; and One or more video items are generated from the image items, the one or more video items comprising a corresponding sequence of one or more image frames, each image frame comprising a portion of the image item selected using the mask, wherein, for each image sequence, the mask is incrementally translated over the image item to select a portion of the image item for consecutive image frames in the sequence.

16. The method as claimed in any of the preceding claims, wherein, The machine learning task includes one or more of the following: object or action detection task, classification task, caption generation task, question answering task, natural language translation task, character or word recognition task, image or audio generation task, or computer language generation task.

17. The method as claimed in any of the preceding claims, wherein, The image item includes images of the document, and the output of the machine learning task depends on the text and / or one or more images in the document.

18. The method of claim 17, wherein, The document is one or more of the following: web page, infographic, form, map, receipt, photographic film, or design drawing of an object or building.

19. The method as claimed in any of the preceding claims, wherein, The image item is an image of a scene comprising multiple objects arranged along a direction corresponding to the second dimension of the image item, and the machine learning task includes identifying one or more of the objects.

20. The method as claimed in any of the preceding claims, wherein, The machine learning task includes an agent control task, wherein the agent interacts with the environment to perform the agent control task, wherein the image item includes observations of the environment, and wherein the output of the machine learning model is used to select one or more actions to be performed by the agent in response to the observations in the environment.

21. The method of claim 20, wherein, The environment described is a real-world environment.

22. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method as claimed in any one of claims 1 to 21.

23. A system comprising: One or more computers; as well as One or more storage devices communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of a corresponding method as described in any one of claims 1 to 21.

Images