Using visual language models to determine the placement of image elements in graphic images

Abstract

A method, implemented by one or more computers, for training a visual language model to identify locations of image elements within an image. The method comprises: generating a plurality of training data elements, each training data element having (i) an image rendered according to a corresponding instruction set, (ii) a natural language query for identifying at least one image element in the image, and (iii) a target location for the at least one image element, the target location being determined from the instruction set. The method further comprises, for each of the training data elements, processing the corresponding image and the natural language query using a visual language model to generate a corresponding model output including a predicted location of an image element identified from the natural language query.The process also includes adjusting parameters of the visual language model to optimize, for each of the training data elements, an objective function that depends on a comparison between the predicted location of the model output corresponding to the training data element and the target location of the training data element. [Figure 1].

Description

Title of the invention: USE OF VISUAL LANGUAGE MODELS TO DETERMINE THE LOCATIONS OF ELEMENTS IMAGE WITHIN GRAPHIC IMAGES

[0001] CONTEXT

[0002] This description relates to the processing of inputs using neural networks to generate output sequences.

[0003] Machine learning models receive an input and generate an output, for example a predicted output, based on the input received. Some machine learning models are parametric models and generate the output based on the input received and the values ​​of the model parameters.

[0004] Some machine learning models are deep models that use multiple model layers to generate an output for a received input. For example, a deep neural network is a deep machine learning model that has an output layer and one or more hidden layers, each of which applies a nonlinear transformation to a received input to generate an output.

[0005] SUMMARY

[0006] This description describes a system implemented as computer programs on one or more computers in one or more locations for training a visual language model to identify the locations of image elements in a graphic image, such as a graph, diagram, or technical drawing. As a result of training, the ability of the visual language model to process and reason about graphic images and text related to graphic images can be improved, for example, so that the performance of the visual language model on other tasks, such as character recognition (e.g., text) in a graphic image, or the extraction of information from the graphic image, is enhanced.

[0007] In a first aspect, the invention relates to a method implemented by one or more computers for training a visual language model to identify locations of image elements within an image. The method comprises generating a plurality of training data elements, each training data element comprising (i) a training input including a graphic image rendered according to a corresponding set of instructions, and a natural language query to identify at least one image element of (i) the graphic image, and (ii) a target location for at least one image element. The target location is determined from the instruction set. The method also includes, for each training data element, processing the corresponding graphic image and the natural language query of the training input using the visual language model to generate a corresponding model output that includes a predicted location of an image element identified from the natural language query. The method further includes tuning parameters of the visual language model to optimize, for each training data element, an objective function that depends on a comparison between the predicted location of the model output corresponding to the training data element and the target location of the training data element.

[0008] As used herein, a graphic image is an image (for example, two-dimensional) composed of a plurality of image elements that provides a visual representation ("visualization") of data, processes, events, or objects. By way of example, a graphic image may include one or more of the following: a graph, such as a bar graph or line graph that provides a visual representation of a particular set of data, for example, in a technical report; a diagram, such as a CAD drawing, for example, for use in manufacturing a product, or an electronic circuit diagram, for example, an integrated circuit layout; and a process flow diagram, for example, for assembling or otherwise configuring a product, or for synthesizing a chemical.In some cases, the graphic image may include text, for example, for labeling, subtitling, or annotating the graphic image, or image elements within the graphic image, for example, to provide axis labels or tick values, a legend or key, a title, etc. In some cases, the graphic image may also or alternatively include one or more raster images (for example, photographic images, bitmaps, or image elements decorated using an image-based texture). As an example, the graphic image may include a flowchart (for example, representing a manufacturing process) that includes one or more photographic images of how the product or components should look (for example, at each manufacturing stage) alongside text (for example, manufacturing instructions).Thus, in some implementations, the visual language model can be trained to identify the locations (e.g., bounding boxes) of image elements that are photographic image elements.

[0009] By way of specific examples, image elements may include one or more of the following: lines, curves, shapes, polygons or other geometric forms, vertices, points, vectors, text, characters, symbols, a rasterized image (for example, a bitmap image), etc. In some cases, image elements may be represented as vector elements before rendering (for example, rasterizing) the graphic image to obtain a corresponding representation of the image elements as pixel values, for example, in a bitmap image. Instructions for rendering a graphic image may, for example, include instructions in a computer programming language, for example, instructions including calls to a computer graphics or plotting library, and / or a markup language (such as HTML, LaTeX, XML, etc.).

[0010] As used here, a visual language model (VLM) is a multimodal language model for processing inputs comprising image and text data modalities. In other words, the VLM can be configured to receive a model input comprising text and image data, and process the model input to generate a corresponding model output. In general, the model output can include data of any modality or all appropriate modalities, for example, text data, image data, a combination of text and image data, and so on. By way of example, the text data can include instructions for a computer, for example, computer code or instructions in one or more programming languages.In some examples, the VLM is configured to process the image as a sequence of image tokens, for example, with text tokens generated from the natural language query. In general, the rendered image processed by the VLM includes pixel data encoding color and / or intensity values ​​for the image pixels.

[0011] In some implementations, the generation of the plurality of training data elements includes, for each of the training data elements: processing the instruction set corresponding to the graphic image of the training data element to generate a data structure comprising one or more sets of coordinates for at least one image element of the training data element (for example, coordinates corresponding to each of the vertices of the image element); and determining the target location of at least one image element using the data structure.

[0012] For example, determining the target location of at least one image element using the data structure may include converting one or more pairs of coordinates into corresponding pixel locations in the graphic image. Thus, in some implementations, the data structure may include a vector representation of the image element(s) from which a rasterized graphic image can be obtained.

[0013] In some implementations, the natural language query may include a natural language description of at least one image element, the description being generated using (i) the instruction set corresponding to the graphic image, or (ii) the data structure, or (iii) both.

[0014] In some implementations, generating the natural language description of at least one image element may involve updating a query template by replacing one or more placeholder elements of the query template with corresponding properties of at least one image element. For example, the query template may include natural language instructions for generating the natural language query.

[0015] By way of example, one or more properties of at least one image element may include one or more of the following: a name of the image element; a type of the image element; a label in the graphic image corresponding to the image element; a shape of the image element; a style, color or texture of the image element; an orientation of the image element; and text associated with the image element.

[0016] In some implementations, the natural language query can be generated by processing the updated query model using a language model, such as a language model neural network. For example, the language model neural network can be a large language model (LLM) neural network. The language model can be the same as the visual language model, or it can be a different language model, for example, a language model that has been refined to generate natural language queries for the particular task of training the visual language model.

[0017] In some implementations, each set of instructions is generated by sampling one or more values ​​of a corresponding image property from a distribution of values ​​for the image property. For example, the distribution of values ​​for the image property may be a discrete distribution, so that the one or more sampled values ​​are each from a predefined set of values, such as styles or fonts for text, or the distribution of values ​​may be a continuous distribution, for example, a uniform or non-uniform distribution for a location within a predefined range or region of the graphic image. In some cases, the one or more sampled values ​​of a corresponding image property may include text, for example, text sampled from a language model, such as an LLM or the VLM. For example, text can be generated by a language model in response to one or more text prompts being provided to the language model.

[0018] For example, one or more image properties may include one or more of the following: an arrangement of image elements in the image; a color or style for the image or for an image element of the image; a size of the image or of an image element of the image; text to be displayed in the image; text to label an image element of the image; and values ​​for quantities represented in a graph of the image.

[0019] In some implementations, each predicted location may include vertex locations of a polygon that surrounds all or part of the corresponding image element. For example, the vertex locations may define a bounding box for the image element, for example, a rectangular or square bounding box.

[0020] In some examples, each predicted location may include one or more pixel locations in the graphic image for the corresponding image element.

[0021] In some implementations, each graphic image includes text (for example, alphanumeric characters). For example, at least one image element may include text, for example a label, a caption, or a reference number for another image element in the graphic image.

[0022] In some cases, each graphic image includes respectively one or more of the following elements: a diagram, a graph, a data table and a map.

[0023] In some implementations, the method also includes, after training the visual language model: receiving an image (for example, a graphic image) and a natural language query to identify at least one image element of the image; and processing the image and the natural language query using the (trained) visual language model to predict a location of the at least one image element of the image identified from the natural language query.

[0024] In certain implementations, the method also includes, after training the visual language model, using the visual language model to perform a character or word recognition task on an image, for example, a graphic image. For example, the visual language model can process a model input comprising one or more images to generate a corresponding model output identifying one or more characters contained in the one or more images. The one or more "characters" can, for example, include alphanumeric characters, logograms (for example, Chinese characters and / or characters from other logographic writing systems), symbols, syllabograms, ideograms, ideographs, pictograms, graphemes, symbols or graphic objects, and the like.

[0025] In another aspect, the invention relates to a method implemented by one or more computers for using a visual language model to identify the locations of image elements within an image, for example, a graphic image. The method comprises receiving an image and a natural language query to identify at least one image element within the image. The method also comprises processing the image and the natural language query using a visual language model, for example, trained using the method described in the first aspect above, in order to predict the location of at least one image element within the image identified from the natural language query.

[0026] In some implementations, the method includes annotating the image at a location derived from the predicted location of at least one image element.

[0027] In some implementations, the method includes generating linking data that associates the predicted location of at least one image element with a corresponding substring of the natural language query or with text output from the VLM. For example, the linking data can be used to align words, phrases, or paragraphs in image-related text (e.g., a summary or description of the image) generated by the VLM with the corresponding image elements. For example, the linking data can be used to create a user interface in which a user can access a relevant figure or image after selecting a substring of the text, or vice versa.

[0028] In some implementations, the method also includes performing an image processing operation on the image based at least on the predicted location of at least one image element. The image processing operation may, for example, include one or more of the following: highlighting at least one image element (for example, by changing a color of the image element, drawing a border around a region including the image element, and so on), cropping the image (for example, to exclude a region of the image that includes, or alternatively does not include, the image element), scaling a region of the image including at least one image element, and deleting at least one image element from the image.For example, once the image processing task has been performed on the image, the updated image can be processed by a machine learning image processing model, for example by the VLM or another VLM, which performs a machine learning task on the updated image. The performance of the machine learning model of... Image processing in performing the machine learning task can be improved by performing the image processing task on the image itself. For example, by cropping the image or enlarging one or more parts of the image, the machine learning model can be guided to, or focus on, the most relevant part(s) of the image.

[0029] In some implementations, the method also includes using the predicted location of at least one image element to determine the truth (factuality) of a statement (for example, a natural language statement) about the image. For example, the image could be a graph showing the respective population sizes of a plurality of cities, such as London, Berlin, Zurich, Toronto, and New York, and the natural language query can identify that one of the cities is more populous than another, for example, "London has a larger population than New York." The VLM can then determine the locations of the image elements in the graph (for example, if the graph is a pie chart, the pie chart segments) that represent the respective population sizes of London and New York, and based on the locations, determine whether the statement is true or false.For example, the locations of image elements can be provided as input to the VLM (or another VLM) to help the VLM reason about the natural language query.

[0030] By way of further example, the method may include selecting a statement from among a plurality of statements concerning the image based at least in part on the predicted location of at least one image element. The method may then include providing the selected statement as output, for example, in response to a natural language query provided by a user. In some cases, the statements may be generated by a language model (for example, the VLM) as potential responses to the natural language query. For example, the VLM may generate a plurality of candidate responses to the query, process each of the candidate responses with the image to identify whether the response is consistent with the image.For example, continuing with the example above, when processing the statement "London has a larger population than New York," the VLM can identify the respective locations of the bars corresponding to New York and London in the bar graph, and then, based on those locations, determine whether the statement is true or false. In cases where no corresponding image element can be identified by the VLM, the VLM can, for example, determine that the statement is false or that additional information is needed. Thus, the trained VLM can generate more appropriate responses to user queries concerning, for example, graphic images.

[0031] By way of general example, the trained VLM can be used to process an image, captured by a camera, that includes a graphic image, for example a photograph or a video frame containing a graphic image, for example on a display surface, such as a whiteboard, a blackboard, or a paper document. For example, the VLM can be used to process graphic images drawn on a whiteboard or paper, for example, during a design or planning meeting, to perform a number of tasks, such as summarizing, answering questions, or providing feedback on the drawn graphic image.

[0032] Using techniques described in this description, the ability of a visual language model to perform tasks related to graphical images (e.g., "data-rich"), such as graphs, diagrams, or maps, can be improved. In particular, training a visual language model to determine the locations of image elements in graphical images can improve the performance of visual language models that have been pre-trained or refined on datasets that consist mainly or exclusively of natural images, for example, photographs.

[0033] For example, the visual language model can learn to associate parts of textual descriptions of graphic images with corresponding image elements within the graphic images, thereby improving the visual language model's ability to reason about graphic images. Thus, the trained visual language model can, for example, be used to perform tasks on complex documents containing text and graphic images, such as scientific articles, CAD drawings, and so on. The visual language model can also help users understand these documents, for example, by enabling the highlighting or annotation of image elements in response to user requests.The visual language model can also, or alternatively, be used, for example, to determine whether a text describing a graphic image is factually correct, for example, on the basis that image elements identified from the text can be found in the graphic image.

[0034] Details of one or more embodiments of the object of this description are set out in the accompanying drawings and the description below.

[0035] Other features, aspects and advantages of the object will become apparent from the description, drawings and claims. Brief description of the drawings

[0036] Fig. 1 shows an example of a training system for training a visual language model.

[0037] Figure 2 shows an example of a training data generator.

[0038] Figure 3 shows an example of a query generator.

[0039] Fig. 4 is an example of a visual language model.

[0040] Figure 5 is a flow diagram of an example of a drive process of a visual language model.

[0041] Fig. 6 is a flow diagram of an example process for using a visual language model to identify locations of image elements in an image.

[0042] The same reference numbers and designations in the different drawings indicate similar elements. DETAILED DESCRIPTION

[0043] Fig. 1 shows an example of a training system 100 for training a visual language model (VLM) 102, for example, a visual language model neural network, to identify locations of image elements in a graphic image 104. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components and techniques described below are implemented.

[0044] The training system 100 trains the VLM 102, in this case a VLM neural network, using a plurality of training data elements 108, which can for example be stored in a database populated using a training data generator 112, such as the training data generator 112 described below in relation to [Fig.2].

[0045] The VLM 102 can process a model input comprising text and one or more images to generate a corresponding model output, which may include text and / or one or more images. In some cases, the model output may include data of a different modality, for example, audio data, instead of or in addition to the text and / or image data. For example, the VLM 102 can process input sequences comprising tokens that each represent natural language text or (part of) an image or video to generate output tokens that each represent natural language or (part of) an image or video.For example, the VLM can encode an image, for instance, as features for each of a set of tokens or patches that cover the image, or as a sequence of visual tokens selected from a visual token vocabulary, or as a representation of distinct objects in the visual input. These visual tokens may, but do not necessarily have to, be interleaved with text tokens processed by the model.

[0046] The VLM 102 can be pre-trained to perform one or more machine learning tasks and then "fine-tuned" using the training system 100. For example, the VLM 102 can be configured to describe an image using natural language, for example, to perform an image captioning task. As another example, the VLM 102 can be configured to perform an image question answering task, for example, by processing input tokens representing an image and text tokens representing a query about the image or a request to modify the image, and generating output tokens representing a response to the query or representing a version of the image that has been modified according to the request. The VLM 102 can generate output tokens representing an image that is generated in response to input tokens providing a visual and / or audio and / or textual description of a desired image.In some implementations, the VLM 102 can also process input data from other modalities, such as audio data. Here are some examples of visual language models with which the techniques described here can be used: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794); and PaLLX (Chen et al. arXiv:2305.18565).

[0047] Each training data element 110 includes a graphic image 104 rendered according to a corresponding set of instructions, for example a chart, diagram, or infographic. For example, the graphic image may include one or more of the following: a flowchart; a bar chart; a histogram; a pie chart; a line graph; a scatter plot; a box plot; a Gantt chart; a Venn diagram; an area chart; a box-and-whisker plot; a bubble chart; a radar chart; a flowchart; etc.As another example, graphic image 104 may include one or more of the following: a floor plan; a geographical map, for example annotated with route information; a circuit diagram; an integrated circuit layout; a technical drawing or diagram, for example a diagram for a scientific article, a patent-style drawing, or a CAD drawing; assembly instructions for a product, for example a piece of furniture; and a chemical structure. Graphic image 104 may also include text, for example in the form of annotations or labels for elements of the image, for example a label for an axis of a graph, or one or more tables, that is, data presented in tabular form.

[0048] In some implementations, the instruction set used to render the graphic image 104 may include instructions in a computer programming language which, when executed by a computer, cause the computer to render the image 104. The instructions may, for example, use one or more graphics or tracing libraries to compose and / or render the graphic image, for example a vector graphics library.

[0049] Each training data element 110 also includes a natural language query 114 that identifies at least one image element of the image. For example, where the image 104 is a graph, the natural language query 114 might include an instruction relating to one of the bars in the graph, for example, "draw a bounding box around the bar on the far left of the graph." As another example, the image element might be a table or part of a table, for example, a row, a column, or an entry in a table. The natural language query 114 might then, for example, include an instruction such as "highlight the entry with the largest value" or "display the coordinates of a bounding box for the second row of the table."

[0050] Each training data element 110 also includes a target location 116 for at least one image element. In general, the target location is determined from the instruction set used to render the image 104, for example, as described below in relation to [Fig. 2]. The target location 116 may, for example, define a polygon, such as a rectangle or a bounding box that surrounds the image element.

[0051] The VLM 102 processes the image 104 and the natural language query 114 to generate a predicted location 118 for at least one image element identified in the natural language query 114. The predicted location 118 can be issued in any form appropriate to the task of identifying the location of the image element in the image 104. For example, the predicted location 118 can specify a polygon, such as a rectangular bounding box, that surrounds the image element, for example, by specifying the respective pixel coordinates for the vertices of the polygon. In general, the predicted location 118 can include text in a natural or computer language that defines a location of the image element.Computer language can be any formal language used to communicate with a computer, for example, a markup language, a command or configuration language, a data exchange language such as JSON, or a programming language. In some cases, the VLM 102 can be configured to generate output text according to a predetermined format, syntax, or scheme, for example, by performing constrained decoding. An example of constraining the VLM 102 output in this way is described in Koo et al., "Automata-based constraints for language model decoding," arXiv:2407.08103 (2024).

[0052] The drive system 100 also includes a drive motor 120 which receives as input the predicted location 118 and the target location 116, and generates as corresponding output gradients 122, with respect to the current parameters (which can be trained) 124 of the VLM 102, of an objective function that depends on a comparison between the predicted location of the model output and the corresponding target location of the image element. The objective function provides a measure of the performance of the VLM 102 in identifying the location of the image element in the image 104. In general, the objective function can be any objective function appropriate to this task. For example, the objective (loss) function can be a least-squares objective function, a cross-entropy objective function, etc. As an example, the objective function can depend on a sum (e.g., of the squares of) the distances between each pair of coordinates in the predicted location 118 and the corresponding pair of coordinates in the target location 116.The gradients 122 with respect to the trainable parameters of the VLM 102 can be determined by backpropagating gradients of the objective function through the layers of the neural network, and then used to update the current parameters 124 of the VLM 102 to obtain updated parameters 126 for use in a subsequent iteration of the training. The training engine 120 can, for example, use a conventional optimizer, such as a stochastic gradient descent optimizer, RMSprop, or Adam.

[0053] For simplicity, [Fig. 1] shows a single predicted location 118 being generated by the VLM 102, but the VLM 102 can, for example, generate a plurality of predicted locations 118, e.g., a respective predicted location 118 for each of a plurality of image elements in the graphic image 104. Thus, each training data element 110 can include a corresponding plurality of target locations 116, and the natural language query 114 can be used to identify a plurality of image elements. By way of example, the natural language query 114 can include "output locations for each of the data points in the point cloud".As an example, the VLM can generate a plurality of predicted locations for the image element, such that each predicted location corresponds to a different respective hypothesis for the image element's location. This can be achieved using a top A sampling algorithm, for example, in which a respective probability is determined for each of the predicted locations as being the correct hypothesis, and a top A (e.g., top-3, -5, -10, etc.) of the predicted locations is selected based on these probabilities. For example, in some cases, the VLM can be trained using a loss-based method in which the respective losses of a top A predicted location are obtained and then combined with a respective weighting determined from the probability that the predicted location for the image element is correct.

[0054] Figure 2 shows an example of a training data generator 112 which processes a set of instructions 202 to generate a data element a corresponding training data structure 110 comprising an image 104, a target location 116, and a natural language query 114. The instructions 202 are processed by a tracing library 204 to generate a corresponding image data structure 206, which is passed to a rendering engine 208, which may be part of or separate from the tracing library 204, to render the image 104. The image data structure 206 is also processed by an image element selector 210, which selects one or more image elements 212 from the image 104, for example, by randomly selecting an image element 212 from among the image elements in the image data structure 206. The training data generator 112 also includes a location extractor 214, which determines the target location 116 of the image element 212 using the image data structure 206 and the selected image element(s) 212.The training data generator 112 also includes a query generator 216 which processes the image data structure 206 and the selected image element(s) 212 to determine the natural language query 114.

[0055] Figure 3 shows an example of a query generator 216 that includes a model engine 302 and a language model 304. The model engine 302 is configured to receive the image element 212 and the image data structure 210 as input, along with a query model 304, to generate a corresponding description 306 of the image element 212 as it will appear in the image 104. For example, the query model 304 may include one or more placeholder elements that are replaced by corresponding properties of the image element 212. The query model 304 may further include natural language instructions to generate the natural language query 106. The language model 304, for example the VLM 102, processes the description 306 to generate the natural language query 106.For example, the 304 language model can be a large language model (LLM) neural network, for instance, a network with over 1 billion, 10 billion, or 100 billion trained parameters. The language model neural network may have been trained on over 10 billion, 100 billion, or 1 trillion words or tokens representing words or other text tokens, such as subwords (also called "word pieces"). In some cases, the VLM 102 can be configured to generate queries according to a predetermined format, syntax, or scheme, for example, by performing constrained decoding. An example of constraining the output of the VLM 102 in this way is described in Koo et al. "Automata-based constraints for language model decoding" arXiv:2407.08103 (2024).

[0056] In some implementations, the 304 language pattern may be omitted. For example, the query pattern may then include language statements natural language to generate the query in natural language, for example, "In the bar chart, locate the bar that is <%=bar.identifier%>", where <%=bar.identifier%> is a placeholder for a string that identifies one of the bars in the bar chart, for example, "smallest", "largest", "red", "shaded", "labeled CO2 emissions for 2011", etc.

[0057] Figure 4 shows an example of a graphic image 104, in this case a bar chart, and a natural language query 114 (here referred to as an "instruction"), which includes the text "draw a bounding box around the bar located on the far left of the chart". Thus, the image element 212 in this case is the leftmost bar of the bar chart. The VLM processes the image 104 and the natural language query 114 to generate a predicted location 112, which in this case is a string comprising "BBOX 80 129 315 219", denoting a rectangular bounding box (BBOX) having, in pixel coordinates, horizontal (x) limits 80 and 219 and vertical (y) limits 315 and 219.

[0058] Figure 5 is a flowchart representing an example of a process 500 for training a Visual Language Model (VLM), such as the VLM 102 described above in relation to Figure 1, in order to identify the locations of image elements within an image. For convenience, the process 500 will be described as being carried out by a system of one or more computers located in one or more locations. For example, a training system, such as the training system 100 of Figure 1, programmed appropriately in accordance with this description, can carry out the process 500.

[0059] The system generates a plurality of training data elements (step 502). Each training data element comprises (i) an image rendered according to a corresponding instruction set, (ii) a natural language query identifying at least one image element of the image, and (iii) a target location for the at least one image element, the target location being determined from the instruction set.

[0060] The system then processes, for each of the training data elements, the corresponding image and natural language query using the VLM in order to generate a corresponding model output including a predicted location of the image element identified in the natural language query (step 504).

[0061] The system then adjusts VLM parameters in order to optimize, for each of the training data elements, an objective function which depends on a comparison between the predicted location of the model output and the corresponding target location (step 506).

[0062] Figure 6 is a flow diagram of an example process 600 allowing the use of a visual language model, such as the VLM 102 described above in relation to with [Fig. 1], for example trained using process 600 of [Fig. 5], in order to identify the locations of image elements within an image, for example a graphic image. For convenience, process 600 will be described as being carried out by a system of one or more computers located in one or more locations.

[0063] The system receives an image and a natural language query to identify at least one image element of the image (step 602).

[0064] The system then processes the image and the natural language query using the visual language model in order to predict a location of at least one image element of the image identified from the natural language query (step 604).

[0065] The system can then (optionally) perform an image processing operation on the image on the basis at least of the predicted location of at least one image element (step 606).

[0066] The trained VLM 102 can, for example, be deployed in an environment that allows users to submit requests to the VLM 102 to process specified model inputs, including one or more images and / or text, to generate corresponding model outputs. Users can submit the requests, for example, through a user interface or via an application programming interface (API). The requests can be transmitted from a user device (for example, over a data communication network, such as the Internet) to one or more computers implementing the VLM 102, for example, in a data center. The VLM 102 can process the model inputs specified by the user requests to generate corresponding model outputs and then transmit the model outputs to the user devices (for example, over a data communication network).

[0067] In general, the VLM 102 can be trained to perform one or more other machine learning tasks, i.e., one or more tasks in addition to the task of predicting the locations of image elements identified from natural language queries. This training can take place before or after the training described above, for example in relation to [Fig. 5]. Once the VLM 102 has been trained, it can then be deployed for use in performing one or more other machine learning tasks.

[0068] In certain implementations, after training, a particular task to be performed by the VLM may be described by part or all of a text sequence in the VLM 102 input template. For example, in an input that includes an image or video element, such a prompt might specify "Generate a caption," "Generate a description," or "Answer the following question: [about the image or video element]." Furthermore, or instead, such a prompt It can provide one or more examples of a task to be performed. A VLM 102 can be trained on multiple natural and / or computer languages, and the prompt can then specify a language to use.

[0069] Here are some examples of other machine learning tasks that can be performed by a VLM trained as described here.

[0070] By way of example, the task may include an object detection task. A task-specific training data element may include an image or video element containing one or more objects and, optionally, a text sequence. The text sequence may describe or otherwise identify the object(s) to be classified. The VLM may provide a model output that includes text giving the bounding box coordinates for the object(s). After training, when the VLM 102 is used for inference, the model output may include or represent text that describes or otherwise labels the detected object(s), and may include the bounding box coordinates for the detected object(s), for example, in the case where the image is a map, "10 20 90 100 river 20 30 100 100 car park".Alternatively, or in addition, the model output may include a modified version of the image annotated to identify detected objects.

[0071] By way of another example, the task may include a classification task, for example, an object classification task. A task-specific training data element may include an image or video element containing one or more objects and text classifying the object(s) in the image. The model output (for example, the text output) may describe or otherwise classify the object(s) into one of a plurality of classes. After training, when the VLM 102 is used for inference, the model output may include data, for example, text, that classifies the object(s) into one of the plurality of classes.

[0072] As another example, the task may include an image or video element describing a task, for example, a subtitling task. A task-specific training data element may then include an image or video element and a text sequence describing the image or video element. After training, when the VLM 102 is used for inference, the model output may include data, for example, text, describing an image or audio element. For example, the model output may provide a caption or description for the image or video element, or it may count objects in the image or video element, or it may provide some other form of description of the image or video element.

[0073] By way of another example, the task may include a question-and-answer task about an image or video. A task-specific training data element may then include an image or video element and a text sequence that describes the image or video element. After training, when the VLM 102 is used in In inference mode, the model output can include data, such as text, that answers a question about the image or video specified in a prompt text sequence. This can be used, for example, to answer questions about a graphic image, such as visual plots and graphs. As an example, the trained VLM 102 can be used to process an image, captured by a camera, that includes a graphic image, such as a photograph, or a video frame containing a graphic image, such as on a display surface, such as a whiteboard, blackboard, or paper document.For example, the VLM 102 can be used to process graphic images drawn or sketched on a whiteboard or paper, for example, during a design or planning meeting, to perform a number of tasks, such as summarizing, answering questions, or providing feedback on the drawn or sketched graphic image.

[0074] By way of another example, the task may include a character or word recognition task, for example, an OCR (Optical Character Recognition) task. A task-specific training data element may then include an image or video element and a text sequence that includes text represented in the image or video. After training, when the VLM 102 is used for inference, the model output may include text that represents characters or words, for example, in natural language. The one or more "characters" may, for example, include alphanumeric characters, logograms (for example, Chinese characters and / or characters from other logographic writing systems), symbols, syllabograms, ideograms, ideographs, pictograms, graphemes, graphic symbols or objects, and the like.For example, the VLM 102 can recognize image elements from a graphic image and generate a model output that describes the graphic image as a sequence of graphic objects.

[0075] As another example, the task may include generating still or animated images. A task-specific training data element may include an image or video element and a text sequence that describes the image or video element. After training, when the VLM 102 is used for inference, the model output may include data for an image or video element, for example, image data defining values ​​for the pixels of a still or animated image, and the text sequence in the model's multimodal input may describe or characterize the image or video element to be generated. For example, the VLM 102 may generate or modify a graphic image comprising one or more image elements, for example, to construct a diagram or graph.

[0076] By way of another example, the task may include a task of generating text in a computer language. A training data element specific to a The task may include an image or video element and a sequence of text in a computer language to generate the image or video element. After training, when the VLM 102 is used for inference, the model output may include text in the computer language or another computer language to generate or render an image or audio element in the second input modality, for example, a web page, a plot, or a graph.

[0077] In another example of a computer language text generation task, a task-specific training data element may include an image or video element and a text sequence in a computer language that performs a task related to the image or video element. For example, a data processing task that involves analyzing the content of the image or video element to provide an analysis result, or, for example, a search to retrieve information related to the content of the image or video element. The computer language in the model output may include a computer language that allows a function to be invoked or one or more external APIs to be called. Simply by way of example, such output may be formatted as a JSON object.As before, the text sequence in the model input can define the task to be performed and may include, for example, an image or video element against which the task must be performed. This could involve manipulating specific types of data that benefit from API access, such as mathematical data, date / time data, scientific data, or recent data that may be subsequent to the model training (accessible via a search function or API). After training, when the VLM 102 is used for inference, the model output can include text in the programming language or another programming language to perform a task, such as the one described above, related to an image, video, or audio element present in the second input modality.The process may then involve using text in computer language to perform the task.

[0078] In general, when the model output includes text, it can be provided in the form of speech representing the text.

[0079] This description uses the term "configured" in relation to computer systems and program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system is equipped with software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or several programs include instructions which, when executed by a data processing device, cause the device to perform the operations or actions.

[0080] Embodiments of the object and functional operations described herein can be implemented in a digital electronic circuit, in tangibly incorporated computer software or firmware, in computer hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of these. Embodiments of the object described herein can be implemented in the form of one or more computer programs, that is, one or more computer program instruction modules encoded on a tangible non-transient storage medium for execution by a data processing device or for controlling its operation.Computer storage media can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial memory device, or a combination of one or more of these. Alternatively or in addition, program instructions can be encoded on an artificially generated propagated signal, for example, an electrical, optical, or electromagnetic signal generated by a machine, which is generated to encode information to be transmitted to a suitable receiving device for execution by a data processing device.

[0081] The term "data processing device" refers to data processing hardware and encompasses all types of data processing devices, apparatus, and machines, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be, or also include, special-purpose logic circuitry, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The device may optionally include, in addition to the hardware, code that creates an execution environment for computer programs, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0082] A computer program, which may also be called or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in an environment In computer science, a program may, but does not have to, correspond to a file in a file system. A program may be stored in a portion of a file that contains other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, subroutines, or parts of code. A computer program may be deployed to run on a single computer or on multiple computers located on the same site or distributed across multiple sites and interconnected by a data communication network.

[0083] In this description, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, several engines may be installed and run on the same computer or computers.

[0084] The processes and logic flows described in this description can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by a special-purpose logic circuit, for example an FPGA or an ASIC, or by a combination of a special-purpose logic circuit and one or more programmed computers.

[0085] Computers adapted for running a computer program may be based on general-purpose or special-purpose microprocessors, or both, or on any other type of central processing unit. Generally, a central processing unit will receive instructions and data from read-only memory (ROM) or random-access memory (RAM), or both. The essential components of a computer are a central processing unit for carrying out or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated into, a special-purpose logic circuit.Typically, a computer will also include, or be operationally coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks. However, a computer does not necessarily have to be equipped with such devices. Furthermore, a computer can be integrated into another device, such as a telephone. mobile, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, for example a USB (Universal Serial Bus) flash drive, to name just a few.

[0086] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, for example, EPROM, EEPROM and flash memory devices; magnetic disks, for example, internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0087] To enable interaction with a user, embodiments of the object described herein can be implemented on a computer having a display device, for example a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, to display information to the user, and a keyboard and a pointing device, for example a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback, for example, visual, auditory, or tactile feedback; and the user input can be received in any form, including acoustic, vocal, or tactile input.Furthermore, a computer can interact with a user by sending documents to a device used by the user and receiving documents from that device; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Additionally, a computer can interact with a user by sending text messages or other forms of messaging to a personal device, such as a smartphone running a messaging application, and receiving reactive messages from the user in return.

[0088] Data processing devices for the implementation of machine learning models may also include, for example, special purpose hardware acceleration units for processing common and computationally intensive parts of training or operating workloads, i.e. inference, related to machine learning.

[0089] Machine learning models can be implemented and deployed using a machine learning framework, for example a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework or an Apache MXNet framework.

[0090] Embodiments of the object described herein can be implemented in a computer system that includes a back-end component, for example, a data server, or a middleware component, for example, an application server, or a front-end component, for example, a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the object described herein, or any combination of one or more of these back-end, middleware, or front-end components. The system components can be interconnected by any form or medium of digital data communication, for example, a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), for example, the Internet.

[0091] A computer system may include clients and servers. A client and a server are generally located remotely from each other and typically interact via a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, a server transmits data, for example, an HTML page, to a user device, for example, in order to display data and receive user input from a user interacting with the device, which acts as a client. Data generated at the user device level, for example, the result of the user interaction, can be received at the server level from the device.

[0092] Although this description contains many specific implementation details, these should not be interpreted as limitations on the scope of an invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features described in this description in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple separate embodiments or in any appropriate sub-combination.Furthermore, although the features may be described above as acting in certain combinations and even initially claimed as such, one or more features of a claimed combination may in some cases be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0093] Similarly, although the operations are represented in the drawings and listed in the claims in a particular order, this should not be construed as requiring that these operations be performed in the specific order indicated or in a sequential order, or that all the illustrated operations be performed, to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various modules and system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or grouped into multiple software products.

[0094] Specific embodiments of the object have been described. Other embodiments fall within the scope of the following claims. For example, the actions referred to in the claims can be performed in a different order and nevertheless produce the desired results. By way of example, the processes described in the accompanying figures do not necessarily require the specific order indicated, or the sequential order, to obtain the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

Demands

1. A method carried out by one or more computers for training a visual language model to identify locations of image elements in a graphic image, the method comprising: generating a plurality of training data elements, each training data element comprising (i) a graphic image rendered according to a corresponding instruction set, (ii) a natural language query for identifying at least one image element in the graphic image, and (iii) a target location for the at least one image element, the target location being determined from the instruction set; for each of the training data elements, processing the corresponding graphic image and the natural language query using a visual language model to generate a corresponding model output comprising a predicted location of an image element identified from the natural language query;and the adjustment of parameters of the visual language model in order to optimize, for each of the training data elements, an objective function that depends on a comparison between the predicted location of the model output corresponding to the training data element and the target location of the training data element.

2. A method according to claim 1, wherein the generation of the plurality of training data elements comprises, for each of the training data elements: processing the instruction set corresponding to the graphic image of the training data element to generate a data structure comprising one or more coordinate pairs for at least one image element of the training data element; and determining the target location of at least one image element using the data structure.

3. A method according to claim 1 or 2, wherein the determination of the target location of at least one image element using the data structure comprises the conversion of one or more pairs of coordinates in corresponding pixel locations in the graphic image.

4. A method according to claim 2 or 3, wherein the natural language query includes a natural language description of at least one image element, the description being generated using (i) the instruction set corresponding to the graphic image, or (ii) the data structure, or (iii) both.

5. A method according to claim 4, wherein the generation of the natural language description of at least one image element comprises updating a query template by replacing one or more placeholder elements of the query template with corresponding properties of at least one image element, the query template comprising natural language instructions for generating the natural language query.

6. A method according to claim 5, wherein one or more properties of at least one image element comprise one or more of the following: an image element name; an image element type; an in-image label corresponding to the image element; an image element shape; an image element style, color, or texture; an image element orientation; and text associated with the image element.

7. A method according to claim 5 or 6, wherein the natural language query is generated by processing the updated query model using a language model.

8. A method according to any prior claim, wherein each set of instructions is generated by sampling one or more values ​​of a corresponding image property from a distribution of values ​​for the image property.

9. A method according to claim 8, wherein the one or more image properties comprise one or more of the following: an arrangement of image elements in the graphic image; a color or style for the graphic image or for an image element of the graphic image; a size of the graphic image or of an image element of the graphic image; text to be displayed in the graphic image; text to label an image element of the graphic image; values ​​for quantities represented in a graph of the graphic image.

10. A method according to any prior claim, wherein each predicted location comprises vertex locations of a polygon that surrounds all or part of the corresponding image element.

11. A method according to any prior claim, wherein each predicted location comprises one or more pixel locations in the image for the corresponding image element.

12. A method according to any prior claim, wherein each graphic image includes text.

13. A method according to any prior claim, wherein each graphic image comprises respectively one or more of the following elements: a diagram, a graph, a data table and a map.

14. A method according to any prior claim, further comprising, after training the visual language model: receiving an image and a natural language query to identify at least one image element from the image; and processing the image and the natural language query using the visual language model to predict a location of at least one image element from the image identified from the natural language query.

15. A method according to any prior claim, also comprising, after training the visual language model, using the visual language model to perform a character or word recognition task on an image.

16. A method carried out by one or more computers and for using a visual language model to identify locations of image elements in an image, the method comprising: receiving an image and a natural language query to identify at least one image element of the image; and processing the image and the natural language query using a visual language model, trained using the method according to any one of the preceding claims, in order to predict a location of the at least one image element of the image identified from the natural language query.

17. A method according to any one of claims 14 to 16, also comprising annotating the image at a location derived from the predicted location of at least one image element.

18. A method according to any one of claims 14 to 17, also comprising generating data associating the predicted location of at least one image element with a corresponding substring of the natural language query or with a text output of the visual language model.

19. A method according to any one of claims 14 to 18, also comprising performing an image processing operation on the image based at least on the predicted location of at least one image element.

20. A method according to claim 19, wherein the image processing operation comprises one or more of the following: highlighting at least one image element, cropping the image, scaling a region of the image comprising at least one image element, deleting at least one image element from the image.

21. A method according to any one of claims 14 to 20, also comprising using the predicted location of at least one image element to determine the veracity of a statement about the image.

22. A method according to any one of claims 14 to 21, also comprising selecting a statement from a plurality of statements about the image based at least in part on the predicted location of at least one image element.

23. System comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to carry out the operations of the respective process according to any one of claims 1 to 22.

24. One or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to carry out the operations of the respective process according to any one of claims 1 to 22.

Images