Transformer-driven differentiable image rasterization for generating vector graphics

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for transformer-driven differentiable image rasterization for generating vector graphics. An example method includes identifying a differentiable image rasterization library that includes a set of mathematically differentiable shape operations for creating a raster image. A set of image embeddings is received for an input image from an image encoder. The set of image embeddings and a prompt are provided to a transformer model. The prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the library. Mappings received from the transformer are provided to the library. A reconstructed image is received from the library that is generated from invoking shape operations of the library in accordance with the mappings. The reconstructed image is compared to the input image to generate reconstruction error data for training the transformer model.

Description

TRANSFORMER-DRIVEN DIFFERENTIABLE IMAGE RASTERIZATION FOR GENERATING VECTORGRAPHICSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 675.931 filed July 26, 2024, the disclosure of which is incorporated herein by reference in its entirety.SUMMARY

[0002] This specification generally relates to transformer-driven differentiable image rasterization for generating vector graphics.BACKGROUND

[0003] Image files may be stored in a variety of formats. A raster image file can include pixel data for pixels that make up an image. The raster image file can include, for example, color and other values for each pixel of an image. Another example image format is a vector graphics format. Vector graphics formats can include instructions for drawing shape primitives such as lines, rectangles, and ellipses.SUMMARY

[0004] One innovative aspect of the subject matter described in this specification is embodied in a method that includes: identifying a differentiable image rasterization library' that includes a set of mathematically differentiable shape operations for creating a raster image, wherein each shape operation accepts a set of parameters; providing an input image to an image encoder; receiving, from the image encoder, a set of image embeddings; providing the set of image embeddings and a prompt to a transformer model, wherein the prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; receiving, from the transformer model, mappings of transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; providing the mappings to the differentiable image rasterization library; receiving, from the differentiable image rasterization library, a reconstructed image generated from invoking shape operations of the differentiableimage rasterization library in accordance with the mappings; comparing the reconstructed image to the input image to generate reconstruction error data; and providing the reconstruction error data to the transformer model to train the transformer model.

[0005] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0006] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. Each image embedding can correspond to a portion of the input image. The transformer model can be a large language model. The transformer embeddings can include transformer tokens that each include a set of transformer token elements. The prompt can prompt the transformer model to generate the mappings by mapping transformer token elements to shape operation parameters. The prompt can prompt the transformer model to generate the mappings by mapping different sets of transformer token elements of a given transformer token to parameters of different shape operations. The prompt can prompt the transformer model to generate the mappings by: identifying a total parameter count of shape operations of the differentiable image rasterization library; linearly projecting transformer token elements of the given transformer token to a dow n sampled vector having a size equal to the total parameter count; and sequentially mapping elements of the down sampled vector to parameters of shape operations of the differentiable image rasterization library. The prompt can prompt the transformer model to generate the mappings by associating an opacify parameter with each shape operation that represents an opacify level used to draw a shape of the shape operation. For a set of shape operations whose parameters have been mapped to different transformer elements of the down sampled vector, the transformer model can learn to assign different opacify levels to the opacify parameter for different shape operations in the set of shape operations. Providing the mappings to the differentiable image rasterization library can include providing higher-visibility mappings that include an opacify parameter that is at least a predetermined opacify threshold and not providing lower-visibility mappings that include an opacity parameter that is below the predetermined opacify threshold. The transformer modelcan be configured to reward solutions that include a lower number of higher- visibility mappings. The transformer model can be iteratively trained until the reconstructed error data corresponds to an error level below a predetermined error threshold. A prediction of a number and type of shape operations as an optimization starting point for the transformer model can be provided to the transformer model. The transformer model can receive the prediction of the number and type of shape operations from a separate shape prediction model. A vector graphics file can be received from the differentiable image rasterization library that describes shape operations performed by the differentiable image rasterization library to generate the reconstructed image. The vector graphics file can be provided, in an inference phase, as an inference output for the input image. Text input for the input image can be received and provided to the transformer model, as appended data appended to the set of image embeddings provided to the transformer model. One or more shape operations of the differentiable image rasterization library can have metadata input parameters for one or more of room type or shape connection information. The reconstruction error data used to train the transformer model can include differences in the reconstructed image and the input image resulting from one or more metadata input parameter values.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description, below. Other potential features and advantages of the disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates an example system for transformer-driven differentiable image rasterization for generating vector graphics.

[0009] FIG. 2 illustrates a flowchart of an example method for transformer-driven differentiable image rasterization for generating vector graphics.

[0010] FIG. 3 is a block diagram of a computing system that can be used in connection with computer-implemented methods described in this specification.DETAILED DESCRIPTION

[0011] In general, the disclosure relates to transformer-driven differentiable image rasterization.

[0012] Different approaches may be used for raster to vector conversions for images (e.g., “vectorization’'), by processing a raster image file (e.g., an image file that includes pixel data) and converting the raster image file into a vector graphic (VG) format (e.g., a format that includes, in a vector form, drawing instructions that when executed can generate a same image as represented by the raster image file).

[0013] Vector graphic formats can provide various advantages over raster image formats. For example, in a context of deep learning for architectural design, an important feature of VG files is that VG files can provide relatively compact text-based representations of an architectural drawing or diagram, as compared to raster image formats. The smaller-sized V G representations can be much more amenable to the training of large deep learning models (e.g., transformerbased architectures such as large language models (LLMs)) than raster images. Furthermore, VG representations are generally more scalable than raster image representations.

[0014] Existing approaches for raster to vector conversion generally apply to a specific, constrained space and are not more generally applicable to generating VG versions of a broad set of raster technical drawings. When training an LLM for general application, a large set of compact training images may be required.

[0015] The solution described herein uses deep learning approaches for raster image to vector conversions. With the solution, a transformer learns how to map transformer outputs to parameters of drawing primitives in a differentiable image rasterization library (DIRL). The solution can be leveraged to create efficient and large training data sets for use in other deep learning models, for example. The solution uses a deep learning-based image vectorization approach that integrates with a transformer to produce an integrated model capable of compact, flexible vectorization of technical drawings such as floor plans and other architectural and structural engineering drawings.

[0016] FIG. 1 illustrates an example system 100 for transformer-driven differentiable image rasterization. The system 100 can process input images, such as an input image 102. The input image 102 may be a floor plan image or some other type of image. The input image 102 can be a raster image.

[0017] An image encoder 104 can process the input image 102 to generate image embeddings 106 for the input image 102. The image encoder 104 may be, for example, a vision transformer.The image embeddings 106 can include a sequence of tokens generated by the image encoder 104 from the input image 102. Each embedding in the image embeddings 106 can correspond to a portion (e.g., “patch”) of the input image 102, for example.

[0018] In some implementations and / or for some inputs, text input may be provided along with the input image 102. The text input may be, for example, a caption for the input image 102. As another example, the text input may be metadata such as a building type that a floor plan input image represents. When text input is provided along with the input image 102, the text data can be appended to the image embeddings 106.

[0019] The image embeddings 106 (and possible appended text input) can be provided to a transformer 108. The transformer 108 can be a large language model that is trained to learn how to vectorize raster images. The image embeddings 106 can be provided by a model trainer 107 to the transformer 108 with prompt 109 that prompts the transformer 108 to generate transformer output embeddings 110 that can be mapped to drawing primitives 111 provided by an image rasterizer 112 for generating an image that matches the input image 102. The transformer 108 can generate the transformer output embeddings 1 10, based on the image embeddings 106 (and possible appended text input) and the prompt 109, in response to the prompt 109. The transformer output embeddings 110 can include a sequence of tokens generated by the transformer 108. As one example, the transformer output embeddings may include two hundred tokens where each token includes two thousand numerical values (e.g.. each token may have a dimension of two thousand).

[0020] The drawing primitives 111 may be part of a DIRL provided by the image rasterizer 112. The drawing primitives 111 can include functions for drawing different types of shapes, such as rectangles, ellipses, curves, etc. Each function of the drawing primitives 111 can accept a set of parameters. For example, the drawing primitives 11 1 include a drawRect function that accepts parametersa drawEllipse function that accepts parametersqj, a drawS pline function that accepts parameters ry, . . . , rk. and a drawPoly (e.g., polygon) function that accepts parametersIn some implementations, the DIRL may include drawing primitives designed, for example, at least in part for building design or another type of specific domain. For example, the drawing primitives 111 may include operations that accept shape metadata such as room type, building type, etc. For instance, a shape operation might accept a shape type of “rectangle”, a width value of five, a length value of ten, a location of a shape center of (0.0.0), and a room type value of “Living Room”.

[0021] In some implementations, shape operations of the DIRL can include parameters that represent graphical information that describes connections of shapes to other shapes. For example, a shape operation may accept parameter values of "‘rectangle”, width=5, length=10, center-point=(0, 0,0), and a connected-to parameter of “{ID=0002, ID=0003” (e.g., where “ID=0002” and “ID=0003” specify identifiers of other shapes to which a shape to be drawn by the shape operation is connected). Such graphical shape connection information can be used later in the process for validating result vectorization. Validation information regarding whether reconstructed images have proper shape connections can be used as training data for the transformer 108. For example, resultant vectorizations that include unexpected, disconnected rooms can be provided as reconstruction error data for the transformer 108.

[0022] The DIRL can be considered differentiable in that each function / operation in the drawing primitives 111 is mathematically differentiable from each other (e.g., based on differences in number and ty pe of parameters). For example, the drawRect function may take seven parameters (e.g., four comer coordinates plus three color values), the drawEllipse function may take seven parameters (e.g.. a center coordinate, a height of a bounding rectangle, a width of a bounding rectangle, an angle of rotation, and three color values), the drawSpline may take, for example, thirteen parameters for defining a requested cubic Bezier spline, etc.

[0023] In general, the model trainer 107 can train the transformer 108 to map the transformer output embeddings 110 onto the parameters of drawing primitives 111 of the image rasterizer 1 12 to preserve differentiability and improve the ability7of the transformer 108 to leam how to construct compact yet expressive VG representations of input images. The model trainer 107 can use one or more different approaches for training the transformer 108 to project the transformer output embeddings 110 onto the input parameters of the drawing primitives 111. For example, if the transformer output embeddings 110 include S' tokens (e.g., 200 tokens) each with a dimension of H (e.g., a dimension of 2000 ), the transformer 108 can be tasked with mapping the SxH values in the transformer output embeddings 110 onto, for example, a count of l total parameters of the drawing primitives of the image rasterizer 112.

[0024] In some implementations, the model trainer 107 can prompt the transformer 108 to treat each token in the transformer output embeddings 110 as a single drawing operation. For example, a first token can be mapped to parameters for a first drawing instruction, a second token can be mapped to parameters for a second drawing instruction, etc.

[0025] In some implementations, the transformer 108 can be prompted to draw all possible shapes by sequentially mapping the elements of a token onto all available functions in the drawing primitives. As an example, suppose the drawing primitives 111 collectively have a total of 37 parameters. The transformer can linearly project a 2000-dimension token into a 37- dimension space and sequentially map the elements of a resulting downsampled vector onto the 37 parameters of the drawing primitives 111.

[0026] For example, first, second, third, and fourth portions of the elements of the downsampled vector can specify the parameters for the drawRect, drawEllipse, drawSpline, and drawPoly functions, respectively. In this example, multiple (e.g., four) drawing primitives and associated parameters may be identified. Different approaches may be used to effectively select one drawing primitive for the token from among the identified drawing operations. In some examples, the transformer 108 can be prompted to add an extra parameter for each available drawing primitive. For instance, for the example with 37 total parameters for four drawing primitives, a new count of 41 total parameters can be used. A SoftMax normalization step can be used to transform the values for the parameters such that they 1) sum to a value of 1.0; and 2) represent a respective probability of selecting each shape as a selected shape of the identified drawing operations.

[0027] Different approaches can be used for selecting a particular drawing primitive based on the respective probabilities. In a first approach, the transformer 108 can interpret the probabilities for the drawing primitives as an alpha color channel for each shape (e.g., representing an opacify which with a respective shape is drawn). Over time, the transformer 108 can learn to strongly favor just one shape I drawing primitive (e.g., by learning to assign nearly all probability mass (e.g.. at or approaching 100%) to a single drawing primitive, based on learning that other approaches result in producing resultant images that do not match well to an original target input image. The transformer 108 can learn to set an opacify7value at or approaching zero for other drawing primitives (e.g., effectively drawing them as invisible with only one drawn shape being visible). The transformer 108 can be trained to not invoke drawing primitives for which an opacify is below a threshold (e.g., with the threshold approaching a zeroopacity).

[0028] In a second approach, the transformer 108 can be prompted to randomly select a single shape / drawing primitive based on the respective probabilities. For example, during a given iteration, probabilities for drawRect, drawEllipse, drawSpline, and drawPoly may be 0.08, 0. 12,0.3. and 0.5, meaning the transformer 108 can select a shape to draw based on an 8%, 12%, 30%, or 50% chance of selecting a rectangle, ellipse, spline, or polygon, respectively. To make this approach differentiable, straight-through estimation or a Gumbel SoftMax layer may be used.

[0029] In some cases, the transformer 108 may leam to reconstruct an image from an unexpected or undesired combination or count (e.g.. high count) of shapes. In some implementations, the transformer 108 is prompted, using a regularization approach, to factor in a penalty value that penalizes higher total used shape counts and / or use of non-preferred shape types, to train the transformer 108 to leam more simple shape drawing solutions. For example, the penalty value may be higher for a solution that uses ellipses and curves rather than rectangles or for using a higher count of shapes rather than a lower count of shapes.

[0030] The model trainer 107 can use different approaches to prompt the transformer 108 for learning how many drawing operations to include in a solution (e.g., to prevent or prompt the transformer 108 to not use too many drawing operations that, while resulting in an accurate image reconstruction, result in a larger than necessary vector graphics file. In some cases, a manual configuration may be used which specifies an optimization starting point of a certain number of each type of shape.

[0031] In some implementations, the model trainer 107 can use a shape type / count predictor 114 which can be a separate model from the transformer 108 which is trained to predict a number and type of shapes that might be needed to reconstruct the input image 102. The shape type / count predictor 114 can be, for example, a vision transformer that is fitted with a regression head that predicts a vector of integers, where each element in the vector is a different shape represented by the drawing primitives 111. The prediction can be included in the prompt 109 that is provided to the transformer 108, as an optimization starting point.

[0032] In implementations in which an opacity parameter is used for each shape, a penalty on complexity can be applied during training that can train the transformer 108 to encourage the use of fewer shapes. Deliberate overparameterization approaches can be used during training in which solutions are generated using substantially more shapes than are needed to reconstruct an image. The transformer 108 can be trained to drop shapes by shifting opacity' towards zero and a regularization penalty can be applied that increases a loss value of a solution proportional to a number of non-transparent shapes that are retained by the transformer 108.

[0033] In implementations in which each token in the transformer output embeddings 110 is mapped onto a single drawing operation, the model trainer 107 can prompt the transformer 108 to select its ow n stopping point (e.g., stopping after processing a particular number of tokens). Shorter sequences of drawing operations can be encouraged by adding a loss component that is proportional to a number of processed tokens.

[0034] After the transformer 108 has mapped the transformer output embeddings 110 to particular parameters of particular functions of the drawing primitives 111 of the image rasterizer 112, the transformer 108 (or the model trainer 107, an inference engine 115, or another component) can invoke the particular functions of the drawing primitives 111 using the particular respective parameters, to generate a reconstructed image 116. The image rasterizer 1 12 (or in some cases the transformer 108) can output a vector graphics file 118 that includes a description of the invoked drawing primitives 111 and corresponding parameters.

[0035] During an inference phase in which the inference engine 115 passes image embeddings 106 for an input image 102 and the prompt 109 to a trained version of the transformer 108, the vector graphics file 118 can be a final, VG-based representation of the input image 102. The inference engine 115 may process a substantial number of input files, using the trained version of the transformer 108 to generate VG versions of each input file. As mentioned, the VG versions of input files may be used for other processes, such as training other LLM models. For instance, when input files are floor plans, as shown in FIG. 1, VG versions of the floor plans can be used to train LLM engines that perform artificial intelligence operations relating to floor plans in particular or building design in general.

[0036] Referring again to a training phase (and / or in some implementations, also during inference), reconstruction error data 122 can be generated by comparing the reconstructed image 116 to the input image 102. In the illustrated example, the reconstructed image 116 has some differences with the input image 102. For instance, some room annotations in the reconstructed image 116 are blurry' and are not at the exact same angles as in the input image 102 and some wall portions in the reconstructed image 116 are only partially drawn.

[0037] The reconstructed error data 122 can be provided to the model trainer 107 which in turn can provide feedback, based on the reconstructed error data 122, to the transformer 108 (e g., in subsequent prompts or in other responses or messages). The feedback can be back propagated to the transformer 108 thereby training the transformer 108 to better select drawing primitivesand corresponding parameters and thereby generating higher quality vector graphic versions of input images during inference.

[0038] FIG. 2 illustrates a flowchart of an example method 200 for transformer-driven differentiable image rasterization for generating vector graphics.

[0039] At 202. a differentiable image rasterization library is identified that includes a set of mathematically differentiable shape operations for creating a raster image, where each shape operation accepts a set of parameters. Some shape operations may accept metadata parameters. Metadata parameters can include room type or shape connection information that indicates to which other shapes a shape to be drawn should be connected.

[0040] At 204. an input image is provided to an image encoder.

[0041] At 206, a set of image embeddings is received from the image encoder. Each image embedding can correspond to a portion of the input image.

[0042] At 208, the set of image embeddings and a prompt are provided to a transformer model. The prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the differentiable image rasterization library. The transformer model can be a large language model. A prediction of a number and type of shape operations can be provided to the transformer model as an optimization starting point for the transformer model. The transformer model can receive the prediction of the number and type of shape operations from a separate shape prediction model. In some examples, text input can be received for the input image and the text input can be appended to the image embeddings that are provided to the transformer model.

[0043] The prompt can prompt the transformer model to generate mappings by mapping transformer token elements to shape operation parameters. The prompt can prompt the transformer model to generate the mappings by mapping different sets of transformer token elements of the given transformer token to parameters of different shape operations. The prompt can prompt the transformer model to generate the mappings by: identifying a total parameter count of shape operations of the differentiable image rasterization library; linearly projecting transformer token elements of the given transformer token to a down sampled vector having a size equal to the total parameter count; and sequentially mapping elements of the down sampled vector to parameters of shape operations of the differentiable image rasterization library.

[0044] The prompt can prompt the transformer model to generate the mappings by associating an opacity parameter with each shape operation that represents an opacity level used to draw ashape of the shape operation. For a set of shape operations whose parameters have been mapped to different transformer elements of the dow n sampled vector, the transformer model can leam to assign different opacity levels to the opacity parameter for different shape operations in the set of shape operations.

[0045] At 210, mappings of transformer embeddings to input parameters of shape operations of the differentiable image rasterization library are received from the transformer model. The transformer embeddings can include transformer tokens that each include a set of transformer token elements.

[0046] At 212, the mappings are provided to the differentiable image rasterization library. Providing the mappings to the differentiable image rasterization library can include providing higher-visibility mappings that include an opacity parameter that is at least a predetermined opacity threshold and not providing lower-visibility mappings that include an opacity parameter that is below the predetermined opacity7threshold. The transformer model can be configured to reward solutions that include a lower number of higher- visibility mappings.

[0047] At 214. a reconstructed image is received, from the differentiable image rasterization library, that is generated from invoking shape operations of the differentiable image rasterization library in accordance with the mappings. A vector graphics file that describes shape operations performed by the differentiable image rasterization library to generate the reconstructed image can be received from the differentiable rasterization library. The vector graphics file can be provided, in an inference phase, as an inference output for the input image.

[0048] At 216, the reconstructed image is compared to the input image to generate reconstruction error data.

[0049] At 218. the reconstruction error data is provided to the transformer model to train the transformer model. The transformer model can be iteratively trained until the reconstruction error data corresponds to an error level below- a predetermined error threshold. In examples in which shape operations include shape connection information, the reconstruction error data can include information regarding disconnected shapes in the reconstruction image that were expected to be connected.

[0050] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

[0051] The term '‘data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0052] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0053] As used in this specification, an “engine,” or “software engine,” refers to a software implemented input / output system that provides an output that is different from the input. Anengine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK’’), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[0054] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0055] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random-access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0056] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g.. EPROM, EEPROM, and flash memory devices; magnetic disks, e g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) monitor, an LCD (liquid crystal display) monitor, or an OLED display, for displaying information to the user, as well as input devices for providing input to the computer, e.g., a keyboard, a mouse, or a presence sensitive display or other surface. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending resources to and receiving resources from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0058] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0059] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.

[0060] An example of one such ty pe of computer is shown in FIG. 3, which show s a schematic diagram of a computer system 300. The system 300 can be used for the operations described in association with any of the computer-implemented methods described previously, according to one implementation. The system 300 includes a processor 310, a memory 320, a storage device 330, and an input / output device 340. Each of the components 310, 320, 330, and 340 are interconnected using a system bus 350. The processor 310 is capable of processing instructions for execution within the system 300. In some implementations, the processor 310 is a single-threaded processor. In some implementations, the processor 310 is a multi -threaded processor. The processor 310 is capable of processing instructions stored in the memory 320 or on the storage device 330 to display graphical information for a user interface on the input / output device 340.

[0061] The memory 320 stores information within the system 300. In some implementations, the memory 320 is a computer-readable medium. In one implementation, the memory 320 is a volatile memory unit. In some implementations, the memory 320 is anon-volatile memory unit.

[0062] The storage device 330 is capable of providing mass storage for the system 300. In one implementation, the storage device 330 is a computer-readable medium. In various different implementations, the storage device 330 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.

[0063] The input / output device 340 provides input / output operations for the system 300. In some implementations, the input / output device 340 includes a keyboard and / or pointing device. In some implementations, the input / output device 340 includes a display unit for displaying graphical user interfaces.

[0064] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0065] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should beunderstood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0066] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0067] What is claimed is:

Claims

CLAIMS1. A computer-implemented method comprising: identifying a differentiable image rasterization library that includes a set of mathematically differentiable shape operations for creating a raster image, wherein each shape operation accepts a set of parameters; providing an input image to an image encoder; receiving, from the image encoder, a set of image embeddings; providing the set of image embeddings and a prompt to a transformer model, wherein the prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; receiving, from the transformer model, mappings of transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; providing the mappings to the differentiable image rasterization library; receiving, from the differentiable image rasterization library, a reconstructed image generated from invoking shape operations of the differentiable image rasterization library' in accordance with the mappings; comparing the reconstructed image to the input image to generate reconstruction error data; and providing the reconstruction error data to the transformer model to train the transformer model.

2. The computer-implemented method of claim 1, wherein each image embedding corresponds to a portion of the input image.

3. The computer-implemented method of claim 1, wherein the transformer model is a large language model.

4. The computer-implemented method of claim 1, wherein the transformer embeddings comprise transformer tokens that each include a set of transformer token elements.

5. The computer-implemented method of claim 4, wherein the prompt prompts the transformer model to generate the mappings by mapping transformer token elements to shape operation parameters.

6. The computer-implemented method of claim 5, wherein the prompt prompts the transformer model to generate the mappings by mapping different sets of transformer token elements of a given transformer token to parameters of different shape operations.

7. The computer-implemented method of claim 6, wherein the prompt prompts the transformer model to generate the mappings by: identifying a total parameter count of shape operations of the differentiable image rasterization library; linearly projecting transformer token elements of the given transformer token to a down sampled vector having a size equal to the total parameter count; and sequentially mapping elements of the down sampled vector to parameters of shape operations of the differentiable image rasterization library.

8. The computer-implemented method of claim 7, wherein the prompt prompts the transformer model to generate the mappings by associating an opacity parameter with each shape operation that represents an opacity level used to draw a shape of the shape operation.

9. The computer-implemented method of claim 8, wherein for a set of shape operations whose parameters have been mapped to different transformer elements of the down sampled vector, the transformer model learns to assign different opacity levels to the opacify parameter for different shape operations in the set of shape operations.

10. The computer-implemented method of claim 9, wherein providing the mappings to the differentiable image rasterization library comprises providing higher- visibility mappings that include an opacify parameter that is at least a predetermined opacify threshold and not providing lower-visibility mappings that include an opacify parameter that is below the predetermined opacify threshold.

11. The computer-implemented method of claim 10. further comprising configuring the transformer model to reward solutions that include a lower number of higher- visibility mappings.

12. The computer-implemented method of claim 1, further comprising iteratively training the transformer model until the reconstructed error data corresponds to an error level below a predetermined error threshold.

13. The computer-implemented method of claim 12, further comprising providing, to the transformer model, a prediction of a number and type of shape operations as an optimization starting point for the transformer model.

14. The computer-implemented method of claim 13, wherein the transformer model receives the prediction of the number and type of shape operations from a separate shape prediction model.

15. The computer-implemented method of claim 1, further comprising receiving, from the differentiable image rasterization library, a vector graphics file that describes shape operations performed by the differentiable image rasterization library to generate the reconstructed image.

16. The computer-implemented method of claim 15, further comprising providing, in an inference phase, the vector graphics file as an inference output for the input image.

17. The computer-implemented method of claim 1, further comprising: receiving text input for the input image; and providing the text input for the input image, to the transformer model, as appended data appended to the set of image embeddings provided to the transformer model.

18. The computer-implemented method of claim 1, wherein one or more shape operations of the differentiable image rasterization library have metadata input parameters for one or more of room type or shape connection information, and wherein the reconstructionerror data used to train the transformer model includes differences in the reconstructed image and the input image resulting from one or more metadata input parameter values.

19. A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: identifying a differentiable image rasterization library that includes a set of mathematically differentiable shape operations for creating a raster image, wherein each shape operation accepts a set of parameters; providing an input image to an image encoder; receiving, from the image encoder, a set of image embeddings; providing the set of image embeddings and a prompt to a transformer model, wherein the prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; receiving, from the transformer model, mappings of transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; providing the mappings to the differentiable image rasterization library; receiving, from the differentiable image rasterization library, a reconstructed image generated from invoking shape operations of the differentiable image rasterization library in accordance with the mappings; comparing the reconstructed image to the input image to generate reconstruction error data; and providing the reconstruction error data to the transformer model to train the transformer model.

20. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: identify ing a differentiable image rasterization library that includes a set of mathematically differentiable shape operations for creating a raster image, wherein each shape operation accepts a set of parameters;providing an input image to an image encoder; receiving, from the image encoder, a set of image embeddings; providing the set of image embeddings and a prompt to a transformer model, wherein the prompt prompts the transformer model to generate transformer embeddings and map the transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; receiving, from the transformer model, mappings of transformer embeddings to input parameters of shape operations of the differentiable image rasterization library; providing the mappings to the differentiable image rasterization library; receiving, from the differentiable image rasterization library, a reconstructed image generated from invoking shape operations of the differentiable image rasterization library in accordance with the mappings; comparing the reconstructed image to the input image to generate reconstruction error data; and providing the reconstruction error data to the transformer model to train the transformer model.

Images