Determining positions of image elements within graphical image using visual language model

Abstract

The disclosure relates to determining positions of image elements within a graphical image using a visual language model. A method performed by one or more computers for training a visual language model to identify locations of image elements within an image. The method includes generating a plurality of training data items, each training data item including an image rendered according to a corresponding set of instructions, a natural language query for identifying at least one image element of the image, and a target location of the at least one image element determined from the set of instructions. For each training data item, the corresponding image and the natural language query are processed using a visual language model to generate a corresponding model output that includes predicted locations of image elements identified from the natural language query. Parameters of the visual language model are adjusted to optimize an objective function for each training data item, the objective function depending on a comparison between a predicted position of the model output corresponding to the training data item and a target position of the training data item.

Description

Technical Field

[0001] This disclosure generally relates to using neural networks to process inputs to generate output sequences. Background Technology

[0002] Machine learning models receive input and generate outputs based on the received inputs, such as predicting outputs. Some machine learning models are parametric models, and they generate outputs based on the received inputs and the values ​​of the model parameters.

[0003] Some machine learning models are deep models, which employ multiple layers to generate outputs from received inputs. For example, a deep neural network is a type of deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0004] This specification describes a system implemented as a computer program on one or more computers at one or more locations for training a visual language model to recognize the location of image elements within graphic images (such as charts, diagrams, or technical drawings). Through training, the visual language model's ability to process and infer graphic images and text associated with them can be improved, for example, thereby enhancing the performance of the visual language model on other tasks such as recognizing characters (e.g., text) in graphic images, or otherwise extracting information about graphic images.

[0005] In a first aspect, a method is provided, executed by one or more computers, for training a visual language model to identify the location of image elements within an image. The method includes generating a plurality of training data items, each training data item including (i) a training input comprising a graphic image rendered according to a corresponding instruction set, and a natural language query for identifying at least one image element of the graphic image, and (ii) a target location of at least one image element. The target location is determined by the instruction set. The method further includes: for each of the training data items, processing the corresponding graphic image and the natural language query of the training input using the visual language model to generate a corresponding model output, the corresponding model output including a predicted location of the image element identified from the natural language query. The method further includes tuning the parameters of the visual language model to optimize an objective function for each of the training data items, the objective function depending on a comparison between the predicted location of the model output corresponding to the training data item and the target location of the training data item.

[0006] As used herein, a graphic image is an image composed of multiple image elements (e.g., two-dimensional) that provides a visual representation (“visualization”) of data, processes, events, or objects. For example, a graphic image may include one or more of the following: charts, such as bar charts or line charts, that provide a visual representation of a particular dataset, for example, as part of a technical report; diagrams, such as CAD drawings, for example, used to manufacture products, or electronic circuit diagrams, such as integrated circuit layouts; and flowcharts of processes, such as used to assemble or otherwise configure products, or synthesize chemical products. In some cases, a graphic image may include: text, such as for labeling, captioning, or otherwise annotating the graphic image; or image elements within the graphic image, such as for providing axis labels or scale values, legends or keys, titles, etc. In some cases, a graphic image may additionally or alternatively include one or more raster images (e.g., photographic images, bitmaps, or image elements decorated with image-based textures). As an example, graphic images can include flowcharts (e.g., representing a manufacturing process) that include one or more photographic images of the appearance a product or component (e.g., at each stage of manufacturing) should have, along with text (e.g., manufacturing instructions). Therefore, in some implementations, a visual language model can be trained to recognize the locations of image elements (e.g., bounding boxes) that are elements of the photographic images.

[0007] As a specific example, image elements may include one or more of the following: lines, curves, shapes, polygons or other geometric forms, vertices, points, vectors, text, characters, symbols, rasterized images (e.g., bitmap images), etc. In some cases, image elements may be represented as vector elements before rendering (e.g., rasterizing) the graphic image to obtain a corresponding representation of the image element as pixel values, such as in a bitmap image. Instructions for rendering the graphic image may include, for example, instructions written in a computer programming language, such as instructions including calls to computer graphics or drawing libraries; and / or instructions written in markup languages ​​(such as HTML, LaTeX, XML, etc.).

[0008] As used herein, a Visual Language Model (VLM) is a multimodal language model used to process inputs that include both image and text data modalities. That is, a VLM can be configured to receive model inputs including text and image data and process those inputs to generate corresponding model outputs. Generally, the model output can include data of any appropriate modality, such as text data, image data, combinations of text and image data, etc. As an example, text data can include instructions for a computer, such as computer code or instructions written in one or more programming languages. In some examples, a VLM is configured to process images as sequences of image lexical units, for example, along with text lexical units generated from a natural language query. Generally, a rendered image processed by a VLM consists of pixel data encoding the color and / or intensity values ​​of the image's pixels.

[0009] In some implementations, generating multiple training data items includes: for each training data item: processing a set of instructions corresponding to the graphic image of the training data item to generate a data structure, the data structure including one or more sets of coordinates (e.g., coordinates corresponding to each vertex of the image element) of at least one image element of the training data item; and using the data structure to determine the target location of at least one image element.

[0010] For example, using a data structure to determine the target location of at least one image element may include converting one or more coordinate pairs into corresponding pixel locations within the graphic image. Therefore, in some implementations, the data structure may include vector representations of image elements from which a rasterized graphic image can be obtained.

[0011] In some implementations, a natural language query may include a natural language description of at least one image element, which is generated using (i) a set of instructions corresponding to the image, or (ii) a data structure, or (iii) both.

[0012] In some implementations, generating a natural language description of at least one image element may include updating the query template by replacing one or more placeholder elements of the query template with the corresponding attributes of at least one image element. As an example, the query template may include natural language instructions for generating the natural language query.

[0013] For example, one or more attributes of at least one image element may include one or more of the following: the name of the image element; the type of the image element; the label corresponding to the image element in the graphic image; the shape of the image element; the style, color, or texture of the image element; the orientation of the image element; and the text associated with the image element.

[0014] In some implementations, natural language queries can be generated by processing updated query templates using a language model, such as a language model neural network. For example, the language model neural network could be a large language model (LLM) neural network. This language model can be the same as the visual language model, or it can be a different language model, such as one fine-tuned to generate natural language queries for a specific task used to train the visual language model.

[0015] In some implementations, each instruction set is generated by sampling one or more values ​​of a corresponding image attribute from a distribution of image attribute values. For example, the distribution of image attribute values ​​can be discrete, such that the sampled values ​​are each of a predefined set of values, such as text style or font; or the distribution can be continuous, for example, a uniform or non-uniform distribution at a location within a predefined range or region of a graphic image. In some cases, the sampled values ​​of a corresponding image attribute can include text, for example, text sampled from a language model such as an LLM or VLM. For example, the language model can generate text in response to one or more text prompts provided to the language model.

[0016] For example, one or more image attributes may include one or more of the following: the arrangement of image elements in the image; the color or style of the image or image elements of the image; the size of the image or image elements of the image; the text displayed in the image; the text used to label the image elements of the image; and the value representing the quantity in the image's chart.

[0017] In some implementations, each predicted location may include the positions of the vertices of a polygon that surrounds all or part of the corresponding image element. For example, the positions of the vertices may define the bounding box of the image element, such as a rectangular or square bounding box.

[0018] In some examples, each predicted location may include one or more pixel locations of the corresponding image element within the graphic image.

[0019] In some implementations, each graphic image includes text (e.g., alphanumeric characters). For example, at least one image element may include text, such as a label, caption, or reference number for another image element in the graphic image.

[0020] In some cases, each graphic image includes one or more of the following corresponding elements: illustration, chart, data table, and map.

[0021] In some implementations, the method further includes, after training a visual language model: receiving an image (e.g., a graphic image) and a natural language query for identifying at least one image element of the image; and using the (trained) visual language model to process the image and the natural language query to predict the location of at least one image element of the image identified from the natural language query.

[0022] In some implementations, the method further includes, after training the visual language model, using the visual language model to perform a character or word recognition task on an image (e.g., a graphic image). For example, the visual language model can process model input including one or more images to generate a corresponding model output that recognizes one or more characters contained in the one or more images. One or more "characters" can include, for example, alphanumeric characters, ideographs (e.g., characters from Chinese characters and / or other ideographic writing systems), symbols, syllabic scripts, logographic scripts, hieroglyphics, graphemes, graphic symbols, or objects.

[0023] In another aspect, a method is provided, executed by one or more computers, for identifying the location of image elements within an image (e.g., a graphic image) using a visual language model. The method includes receiving an image and a natural language query for identifying at least one image element of the image. The method further includes processing the image and the natural language query using a visual language model to predict the location of at least one image element of the image identified from the natural language query, the visual language model being trained, for example, using the method of the first aspect described above.

[0024] In some implementations, the method includes annotating the image at a location derived from the predicted location of the at least one image element.

[0025] In some implementations, the method includes generating linking data that associates the predicted location of the at least one image element with a corresponding substring of a natural language query or the text output of a VLM. For example, the linking data can be used to align words, sentences, or paragraphs in image-related text (e.g., a summary or description of the image) generated by the VLM with corresponding image elements of the image. For example, the linking data can be used to create a user interface in which a user can navigate to a related graphic or image after selecting a substring of text, and vice versa.

[0026] In some implementations, the method further includes performing image processing operations on the image based at least on the predicted position of the at least one image element. Image processing operations may include, for example, one or more of the following: highlighting at least one image element (e.g., by changing the color of the image element, drawing a border around the region including the image element, etc.), cropping the image (e.g., to exclude regions of the image that include or alternatively do not include the image element), scaling regions of the image that include at least one image element, and removing at least one image element from the image. For example, after performing image processing tasks on the image, the updated image can be processed by an image processing machine learning model (e.g., by a VLM or another VLM) that performs machine learning tasks on the updated image. Performing image processing tasks on the image can improve the performance of the image processing machine learning model in performing machine learning tasks. For example, by cropping or enlarging portions of the image, the machine learning model can be directed to or focused on more relevant parts of the image.

[0027] In some implementations, the method further includes using the predicted location of at least one image element to determine the veracity (factuality) of a statement about the image (e.g., a natural language statement). For example, the image could be a chart indicating the corresponding population size of each of several cities (such as London, Berlin, Zurich, Toronto, and New York), and a natural language query could identify that one of these cities has a larger population than another, such as "London has a greater population than New York." The VLM can then determine the location of image elements in the chart depicting the corresponding population sizes of London and New York (e.g., sectors of a pie chart if the chart is one), and based on these locations, determine whether the statement is true or false. For example, the location of the image element can be provided as input to the VLM (or another VLM) to assist the VLM in reasoning about the natural language query.

[0028] As another example, the method may include selecting a statement from multiple statements about an image, at least in part based on the predicted location of at least one image element. The method may then include providing the selected statement as output, for example, in response to a natural language query provided by a user. In some cases, these statements may be generated by a language model (e.g., a Virtual Language Model) as potential responses to a natural language query. For example, a VLM may generate multiple candidate responses for a query, processing each of the candidate responses along with an image to identify whether the response is consistent with the image. For example, continuing the example above, when processing the statement "London has a greater population than New York," the VLM may identify the corresponding locations of bars in a histogram that correspond to New York and London, and then determine whether the statement is true or false based on these locations. If the VLM cannot identify any corresponding image element, the VLM may, for example, determine that the statement is false, or that more information is needed. Therefore, a trained VLM can generate a more appropriate response to a user query about, for example, a graphical image.

[0029] As a general example, a trained VLM can be used to process images captured by a camera, including graphic images, such as photographs or video frames containing graphic images, for example, on a display surface such as a whiteboard, blackboard, or paper document. For example, a VLM can be used to process graphic images drawn on a whiteboard or paper, for example, during a design or planning meeting, to perform any of a number of tasks, such as summarizing, answering questions, or otherwise providing feedback on the drawn graphic images.

[0030] Using the techniques described in this specification, the ability of visual language models to perform tasks related to graphical (e.g., “data-intensive”) images (such as charts, diagrams, or maps) can be improved. Specifically, training visual language models to determine the location of image elements in graphical images can improve the performance of visual language models pre-trained or fine-tuned on datasets that primarily include or only include natural images (e.g., photographs).

[0031] For example, a visual language model can learn to associate portions of a textual description of a graphic image with corresponding image elements within the image, thereby enhancing its ability to reason about graphic images. Therefore, a trained visual language model can be used, for example, to perform tasks on complex documents that include both text and graphic images (such as scientific articles, CAD drawings, etc.). Visual language models can also help users understand such documents, for example, by allowing the highlighting or annotation of image elements in response to user requests. Visual language models can also be used, for example, to determine whether the text describing a graphic image is factually accurate, for instance, based on whether image elements recognizable from the text can be found within the graphic image.

[0032] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description.

[0033] Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. Attached Figure Description

[0034] Figure 1 An example training system for training a visual language model is shown.

[0035] Figure 2 An example training data generator is shown.

[0036] Figure 3 An example query generator is shown.

[0037] Figure 4 This is an example visual language model.

[0038] Figure 5 This is a flowchart of an example process for training a visual language model.

[0039] Figure 6 This is a flowchart of an example process for using a visual language model to identify the location of image elements within an image.

[0040] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0041] Figure 1 An example training system 100 is shown for training a visual language model (VLM) 102 (e.g., a visual language model neural network) to identify the location of image elements within a graphic image 104. The training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the systems, components and techniques described below are implemented.

[0042] Training system 100 uses multiple training data items 108 to train VLM 102 (in this case, a VLM neural network), which can be stored, for example, using training data generator 112 (such as the combination below). Figure 2 The training data generator (112) described in the description populates the database.

[0043] The VLM 102 can process model inputs including text and one or more images to generate corresponding model outputs, which may include text and / or one or more images. In some cases, the model output may include data of different modalities, such as audio data, instead of text and / or image data. For example, the VLM 102 can process input sequences including lexical units, each representing a portion of natural language text, an image, or video, to generate output lexical units, each representing a portion of natural language, an image, or video. For example, the VLM can encode images, for instance, by encoding an image as features that divide the image into a set of lexical units or image patches, or by encoding an image as a sequence of visual lexical units selected from a visual lexical vocabulary, or by encoding an image as representations of different objects in the visual input. Such visual lexical units may (but do not necessarily) interleave with text lexical units processed by the model.

[0044] VLM 102 can be pre-trained to perform one or more machine learning tasks and then "fine-tuned" using training system 100. For example, VLM 102 can be configured to describe images using natural language, for example, to perform an image captioning task. As another example, VLM 102 can be configured to perform an image question-answering task, for example, by processing input tokens representing an image and text tokens representing a query about the image or a request to modify the image, and generating output tokens representing the query answer or a version of the image that has been modified according to the request. VLM 102 can generate output tokens representing an image generated in response to input tokens providing a visual and / or audio and / or text description of the desired image. In some implementations, VLM 102 can also process input data of other modalities, such as audio data. Some example visual language models that can use the techniques described in this paper include: Flamingo (Alayrac et al., arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al., arXiv:2209.06794); and PaLI-X (Chen et al., arXiv:2305.18565).

[0045] Each training data item 110 includes a graphical image 104 rendered according to a corresponding instruction set, such as a chart or graph, diagram, or infographic. For example, the graphical image may include one or more of the following: flowchart; bar chart; histogram; pie chart; line chart; scatter plot; box plot, Gantt chart; Venn diagram; area chart; box-and-whisker diagram; bubble chart; radar chart; flowchart, etc. As another example, the graphical image 104 may include one or more of the following: floor plan; such as a geographical map annotated with route information; circuit diagram; integrated circuit layout; blueprints or technical drawings, such as illustrations from scientific articles, patent drawings, or CAD drawings; assembly instructions for a product (e.g., a piece of furniture); and chemical structures. The graphical image 104 may additionally include text, such as annotations or labels as image elements, such as labels for chart axes, or one or more tables, i.e., data presented in tabular form.

[0046] In some implementations, the instruction set for rendering the graphic image 104 may include instructions in a computer programming language that, when executed by the computer, cause the computer to render the image 104. For example, these instructions may use one or more graphics or drawing libraries to compose and / or render the graphic image, such as a vector graphics library.

[0047] Each training data item 110 further includes a natural language query 114 for recognizing at least one image element of the image. As an example, when image 104 is a chart, natural language query 114 may include instructions related to one of the bars in the chart, such as “draw a bounding box around the bar located at the far left of the chart”. As another example, the image element may be a table or part of a table, such as a row, column, or table entry. Natural language query 114 may then include instructions such as “highlight the entry with the largest value” or “output the coordinates of a bounding box for the second row of the table”.

[0048] Each training data item 110 also includes a target location 116 for at least one image element. Generally, the target location is determined by the instruction set used to render the image 104, for example, as described below. Figure 2 The target location 116 can, for example, define a polygon, such as a "bounding box" or rectangle that surrounds the image element.

[0049] VLM 102 processes image 104 and natural language query 114 to generate a predicted location 118 for at least one image element identified in the natural language query 114. The predicted location 118 can be output in any form suitable for the task of identifying the location of an image element within image 104. For example, the predicted location 118 can specify the polygon, such as a rectangular bounding box, by specifying the corresponding pixel coordinates of the vertices of the polygon surrounding the image element. Generally, the predicted location 118 can include text in natural or computer language defining the location of the image element. The computer language can be any form of language used for communicating with a computer, such as a markup language, or a command or configuration language, or a data exchange language (such as JSON), or a programming language. In some cases, VLM 102 can be configured to generate output text according to a predetermined format, syntax, or pattern (e.g., by performing constraint decoding). Koo et al. describe an example of constraining the output of VLM 102 in this way in “Automata-based constraints for language model decoding” arXiv:2407.08103 (2024).

[0050] The training system 100 further includes a training engine 120 that receives the predicted position 118 and the target position 116 as inputs and generates a gradient 122 of the objective function with respect to the current (trainable) parameters 124 of the VLM 102 as the corresponding output. This objective function depends on the comparison between the predicted position output by the model and the corresponding target position of the image element. The objective function provides a metric for the performance of the VLM 102 in recognizing the position of an image element within the image 104. Generally, the objective function can be any objective function suitable for the task. For example, the objective (loss) function could be a least-squares objective function, a cross-entropy objective function, etc. As an example, the objective function could depend on the sum (e.g., squared) of the distances between each coordinate pair in the predicted position 118 and the corresponding coordinate pair in the target position 116. The gradient 122 of the trainable parameters of the VLM 102 can be determined by backpropagating the gradient of the objective function through the layers of the neural network, and then using these gradients to update the current parameters 124 of the VLM 102 to obtain updated parameters 126 for use in subsequent training iterations. The training engine 120 can use conventional optimizers, such as stochastic gradient descent, RMSprop, or Adam optimizer.

[0051] For the sake of simplicity, Figure 1The diagram shows a single predicted location 118 generated by VLM 102, but VLM 102 can, for example, generate multiple predicted locations 118, such as generating a corresponding predicted location 118 for each of multiple image elements in the graphic image 104. Therefore, each training data item 110 can include multiple corresponding target locations 116, and a natural language query 114 can be used to identify multiple image elements. As an example, the natural language query 114 can include "output locations for each of the data points in the scatter plot". As an example, VLM can generate multiple predicted locations for image elements, such that each predicted location corresponds to a different corresponding hypothesis for the location of the image element, for example, using a top-k sampling algorithm, where a corresponding probability is determined for each predicted location as a correct hypothesis, and the top k (e.g., top 3, top 5, top 10, etc.) locations of the predicted locations are selected based on the probability. For example, in some cases, a loss can be used to train the VLM, where corresponding losses for the top k predicted locations are obtained, and these corresponding losses are then combined with corresponding weights determined based on the probability that the predicted location of the image element is correct.

[0052] Figure 2 An example training data generator 112 is shown that processes instruction set 202 to generate corresponding training data items 110, which include image 104, target location 116, and natural language query 114. Instructions 202 are processed by a graphics library 204 to generate a corresponding image data structure 206, which is passed to a rendering engine 208 to render image 104. The rendering engine may be part of or separate from the graphics library 204. Image data structure 206 is also processed by an image element selector 210, which selects one or more image elements 212 within image 104, for example, by randomly selecting image elements 212 from the image elements of image data structure 206. Training data generator 112 further includes a location extractor 214 that uses image data structure 206 and the selected image elements 212 to determine the target location 116 of image element 212. The training data generator 112 further includes a query generator 216 that processes the image data structure 206 and the selected image elements 212 to determine a natural language query 114.

[0053] Figure 3An example query generator 216 is shown, which includes a template engine 302 and a language model 304. The template engine 302 is configured to receive image element 212 and image data structure 210 along with the query template 304 as input to generate a corresponding description 306 that the image element 212 will appear in image 104. For example, the query template 304 may include one or more placeholder elements replaced by corresponding attributes of image element 212. The query template 304 may additionally include natural language instructions for generating the natural language query 106. The language model 304 (e.g., VLM 102) processes the description 306 to generate the natural language query 106. For example, the language model 304 may be a large language model (LLM) neural network, such as a neural network with more than 1 billion, 10 billion, or 100 billion training parameters. Language model neural networks may have been trained on more than 10 billion, 100 billion, or 1 trillion words or word units representing words or other text units (e.g., subwords (also known as "chunks")). In some cases, VLM 102 can be configured to generate queries based on a predetermined format, grammar, or pattern (e.g., by performing constraint decoding). Koo et al. describe an example of constraining VLM102 output in this way in "Automata-based constraints for language model decoding" arXiv:2407.08103 (2024).

[0054] In some implementations, the language model 304 can be omitted. As an example, the query template can then include natural language instructions for generating natural language queries, such as, “In the bar chart, locate the bar thatis <%=bar.identifier%>”, where <%=bar.identifier%> is a placeholder string that allows identification of one of the bars in the bar chart, such as “smallest”, “largest”, “red”, “shaded”, “labeled CO2 emissions for 2011”, etc.

[0055] Figure 4An example graphic image 104 (in this case, a bar chart) and a natural language query 114 (referred to here as the "instruction") are shown, which includes the text "draw a bounding box around the bar located at the far left of the chart". Therefore, in this case, image element 212 is the leftmost bar of the bar chart. The VLM processes image 104 and the natural language query 114 to generate a predicted location 112, in this case, a string including "BBOX 80 129 315 219", representing a rectangular bounding box (BBOX) with horizontal (x) limits of 80 and 219 and vertical (y) limits of 315 and 219 in pixel coordinates.

[0056] Figure 5 It is used to train visual language models (VLM) (such as those mentioned above). Figure 1 The flowchart 500 describes an example process of using a VLM (102) to identify the location of image elements within an image. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to this specification (e.g., Figure 1 The training system 100) can execute process 500.

[0057] The system generates multiple training data items (step 502). Each training data item includes (i) an image rendered according to a corresponding instruction set, (ii) a natural language query to identify at least one image element of the image, and (iii) the target location of the at least one image element, which is determined by the instruction set.

[0058] The system then uses VLM to process the corresponding image and natural language query for each of the training data items to generate a corresponding model output, which includes the predicted location of the image element identified in the natural language query (step 504).

[0059] The system then adjusts the parameters of the VLM to optimize the objective function for each training data item, which depends on the comparison between the predicted location of the model output and the corresponding target location (step 506).

[0060] Figure 6 It is used to employ visual language models (such as those combined above). Figure 1 The described VLM 102, for example, uses Figure 5The flowchart 600 is an example of a process for training (process 600) to identify the location of image elements within an image (e.g., a graphic image). For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations.

[0061] The system receives an image and a natural language query for identifying at least one image element of the image (step 602).

[0062] The system then uses a visual language model to process the image and the natural language query to predict the location of at least one image element identified from the natural language query (step 604).

[0063] The system can then (optionally) perform image processing operations on the image based on at least the predicted location of at least one image element (step 606).

[0064] The trained VLM 102 can be deployed, for example, in an environment that allows a user to provide the VLM 102 with requests for processing specified model inputs, including one or more images and / or text, to generate corresponding model outputs. The user can provide the requests, for example, through a user interface or application programming interface (API). These requests can be transmitted from the user device (e.g., via a data communication network, such as the Internet) to one or more computers (e.g., in a data center) implementing the VLM 102. The VLM 102 can process the model inputs specified by the user request to generate corresponding model outputs, and then transmit the model outputs to the user device (e.g., via a data communication network).

[0065] Generally, VLM 102 can be trained to perform one or more other machine learning tasks, i.e., one or more tasks other than predicting the location of image elements identified from natural language queries. This training can occur, for example, in conjunction with... Figure 5 The description refers to the period before or after training. After training, the VLM 102 can be deployed to perform one or more other machine learning tasks.

[0066] In some implementations, after training, the specific task to be performed by the VLM can be described by part or all of the text sequence in the VLM 102's model input. For example, in an input that includes an image or video item, such a prompt might specify "Generate a caption," "Generate a description," or "Answer the following question: [about the image, or video item]." Alternatively, or conversely, such a prompt might provide one or more examples of the task to be performed. The VLM 102 can be trained against multiple natural and / or computer languages, and the prompt can then specify the language to be used.

[0067] Here are a few examples of other machine learning tasks that can be performed by a VLM trained as described in this article.

[0068] As an example, the task may include an object detection task. Task-specific training data items may include images or videos containing one or more objects, and optionally, text sequences. The text sequences may describe or otherwise identify the objects to be classified. The VLM can provide model output that includes text giving the bounding box coordinates of the objects. After training, when the VLM 102 is used for inference, the model output may include or represent text describing or otherwise labeling the detected objects, and may include the bounding box coordinates of the detected objects, such as in the case where the image is a map, “10 20 90 100 river 20 30 100 100 car park (10 20 90 100 river 20 30 100 100 parking lot)”. Alternatively or additionally, the model output may include a modified version of the image annotated to identify the detected objects.

[0069] As another example, the task can include a classification task, such as an object classification task. Task-specific training data items can include images or videos containing one or more objects, and text classifying the objects in the images. The model output (e.g., text output) can describe the objects or otherwise classify them into one of multiple categories. After training, when the VLM 102 is used for inference, the model output can include data classifying objects into one of multiple categories, such as text.

[0070] As another example, the task could include an image or video item description task, such as a captioning task. Task-specific training data items could then include image or video items along with a sequence of text describing those items. After training, when the VLM 102 is used for inference, the model output could include data describing the image or audio items, such as text. For example, the model output could provide captions or descriptions for image or video items, count objects within an image or video item, or provide other forms of description for the image or video item.

[0071] As another example, the task could include an image or video question-answering task. Task-specific training data items could then include image or video items along with a sequence of text describing those images or videos. After training, when the VLM 102 is used for inference, the model output could include data, such as text, to answer questions about the image or video specified in the text prompt sequence. For example, this could be used to answer questions about graphic images (e.g., visual graphs and charts). As an example, a trained VLM 102 could be used to process images captured by a camera, including graphic images, such as photographs or video frames of graphic images displayed on surfaces like whiteboards, blackboards, or paper documents. For example, the VLM 102 could be used, for example, to process graphic images drawn or sketched on a whiteboard or paper during a design or planning meeting to perform any of a number of tasks, such as summarizing, answering questions, or otherwise providing feedback on the drawn or sketched graphic images.

[0072] As another example, the task could include a character or word recognition task, such as an OCR (Optical Character Recognition) task. Task-specific training data items could then include image or video items and a sequence of text including the text depicted in the image or video. After training, when the VLM 102 is used for inference, the model output could include text representing, for example, characters or words in natural language. One or more “characters” could include, for example, alphanumeric characters, ideographs (e.g., characters from Chinese characters and / or other ideographic writing systems), symbols, syllabic scripts, logographic scripts, hieroglyphics, graphemes, graphic symbols, or objects. For example, the VLM 102 could recognize image elements of a graphic image and generate a model output that describes the graphic image as a sequence of graphic objects.

[0073] As another example, the task can include a static or moving image generation task. Task-specific training data items can include image or video items and a text sequence describing those image or video items. After training, when the VLM 102 is used for inference, the model output can include data that includes image or video items, such as image data defining pixel values ​​for static or moving images, and the text sequence in the model's multimodal input can describe or characterize the image or video item to be generated. For example, the VLM 102 can generate or modify graphical images that include one or more image elements, for example, to construct diagrams or charts.

[0074] As another example, the task could include a computer language text generation task. Task-specific training data items could include image or video items and a sequence of text in computer language used to generate those image or video items. After training, when the VLM 102 is used for inference, the model output could include text in that computer language or another computer language used to generate or render image or audio items (e.g., web pages, graphs, or charts) from a second modality input.

[0075] In another example of a computer language text generation task, task-specific training data items may include image or video items and a sequence of text in computer language for performing tasks related to the image or video items, such as analyzing the content of the image or video items to provide analysis results or providing a search to search for information related to the content of the image or video items. The computer language in the model output may include computer language used to invoke functions or call one or more external APIs. As just an example, such output may be formatted as a JSON object. As previously mentioned, the text sequence in the model input may define the task to be performed and may include, for example, image or video items related to the task to be performed, such as tasks involving manipulating specific types of data that can benefit from accessing APIs, such as mathematical data, date / time related data, scientific data, recent data that may lag behind the model training time (accessible via search functions or APIs), etc. After training, when the VLM 102 is used for inference, the model output may include text in that computer language or another computer language for performing tasks, such as those related to the image, video, or audio items in the second modality input as described above. The method can then include performing the task using text presented in a computer language.

[0076] Generally, when the model output includes text, it can be provided as speech representing the text.

[0077] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0078] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) to generate artificially generated propagation signals to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0079] The term "data processing device" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0080] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0081] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0082] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0083] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or from them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0085] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0086] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0087] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0088] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0089] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs running on respective computers and establishing a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0090] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0091] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in an ordered sequence, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0092] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions stated in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method executed by one or more computers for training a visual language model to identify the location of image elements within a graphic image, the method comprising: Multiple training data items are generated, each training data item including: (i) a graphic image rendered according to a corresponding instruction set, (ii) a natural language query for identifying at least one image element of the graphic image, and (iii) the target location of the at least one image element, the target location being determined from the instruction set; For each training data item, a visual language model is used to process the corresponding image and natural language query to generate a corresponding model output, the corresponding model output including the predicted location of image elements identified from the natural language query; and The parameters of the visual language model are adjusted to optimize the objective function for each of the training data items, the objective function depending on the comparison between the predicted position of the model output corresponding to that training data item and the target position of that training data item.

2. The method as described in claim 1, wherein, Generating the plurality of training data items includes, for each of the training data items: Process the instruction set corresponding to the graphic image of the training data item to generate a data structure including one or more coordinate pairs of the at least one image element of the training data item; and The data structure is used to determine the target location of the at least one image element.

3. The method as described in claim 1, wherein, Determining the target location of the at least one image element using the data structure includes converting the one or more coordinate pairs into corresponding pixel locations within the graphic image.

4. The method of claim 2, wherein, The natural language query includes a natural language description of the at least one image element, the description being generated using (i) the instruction set corresponding to the graphic image, or (ii) the data structure, or (iii) both.

5. The method of claim 4, wherein, Generating the natural language description of the at least one image element includes updating the query template by replacing one or more placeholder elements of the query template with corresponding attributes of the at least one image element, the query template including natural language instructions for generating the natural language query.

6. The method of claim 5, wherein, The one or more attributes of the at least one image element include one or more of the following: the name of the image element; the type of the image element; the label in the image corresponding to the image element; the shape of the image element; the style, color, or texture of the image element; the orientation of the image element; and the text associated with the image element.

7. The method of claim 5, wherein, The natural language query is generated by processing the updated query template using a language model.

8. The method of claim 1, wherein, Each instruction set is generated by sampling one or more values ​​of the image attribute from the distribution of values ​​of the corresponding image attribute.

9. The method of claim 8, wherein, The one or more image attributes include one or more of the following: the arrangement of image elements in the graphic image; the color or style of the graphic image or the image elements of the graphic image; the size of the graphic image or the image elements of the graphic image; text for display in the graphic image; text for labeling the image elements of the graphic image; and the value of the quantity represented in the chart of the graphic image.

10. The method of claim 1, wherein, Each predicted location includes the positions of the vertices of the polygon that surrounds all or part of the corresponding image element.

11. The method of claim 1, wherein, Each predicted location includes one or more pixel locations of the corresponding image element within the image.

12. The method of claim 1, wherein, Each graphic image includes text.

13. The method of claim 1, wherein, Each graphic image includes one or more of the following: illustrations, charts, data tables, and maps.

14. The method of claim 14, further comprising, after training the visual language model: Receive an image and a natural language query for identifying at least one image element of the image; and The visual language model is used to process the image and the natural language query to predict the location of at least one image element identified from the natural language query.

15. The method of any one of claims 1 to 14, further comprising, after training the visual language model, using the visual language model to perform a character or word recognition task on an image.

16. A method executed by one or more computers for identifying the location of image elements within an image using a visual language model, the method comprising: Receive an image and a natural language query for identifying at least one image element of the image; as well as A visual language model is used to process the image and the natural language query to predict the location of at least one image element identified from the natural language query, the visual language model being trained using the method described in any one of claims 1 to 15.

17. The method of claim 16, further comprising annotating the image at a location derived from the predicted location of the at least one image element.

18. The method of claim 16, further comprising generating data that associates the predicted location of the at least one image element with a corresponding substring of the natural language query or the text output of the visual language model.

19. The method of claim 16, further comprising performing image processing operations on the image based at least on the predicted position of the at least one image element.

20. The method of claim 19, wherein, The image processing operations include one or more of the following: highlighting the at least one image element, cropping the image, scaling the region of the image that includes the at least one image element, and removing the at least one image element from the image.

21. The method of claim 16, further comprising using the predicted location of the at least one image element to determine the veracity of a statement about the image.

22. The method of any one of claims 16 to 21, further comprising selecting a statement from a plurality of statements about the image based at least in part on the predicted location of the at least one image element.

23. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform the operations of the method as described in any one of claims 1 to 22.

24. A computer storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method as described in any one of claims 1 to 22.

Images