Method and system for generating vector graph based on text of large language model

Abstract

The invention relates to the technical field of image generation, in particular to a method and system for generating a vector graph based on a text of a large language model, and the method comprises the steps: carrying out the optimization of collected SVG data; rendering the optimized SVG data to obtain a corresponding rasterized image, using a multi-modal large model to generate a corresponding label for the rasterized image as a basic text description, and constructing a multi-modal data set; each entry in the multi-modal data set is a triple composed of the optimized SVG data, the corresponding rasterized image and the basic text description; the SVG data and the basic text description are input into a GPT large model, and detailed text description for the SVG data is generated; and combining the SVG data, the corresponding rasterized image and detailed text description with the collected instruction data set to serve as a training set, and training a pre-constructed basic large language model under the understanding task and the generation task respectively. According to the method, the fine SVG vector graph with consistent semantics can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and more particularly to a method and system for generating vector graphics from text based on a large language model. Background Art

[0002] Scalable Vector Graphics (SVG), an XML-based image format, is widely used in modern digital design, such as web design, user interfaces, icons, and emoticons, due to its resolution independence, high storage efficiency, and ease of editing. However, current text-to-SVG generation technologies face the following problems and limitations:

[0003] 1. Lack of high-quality vector graphics datasets: The datasets suitable for vector graphics generation in existing technologies are limited in size and cannot support complex graphics generation tasks.

[0004] 2. Low generation efficiency of existing methods: Optimization-driven methods usually rely on differentiable renderers, and their optimization process is computationally intensive and time-consuming.

[0005] 3. Limited applicability of language model methods: Technologies that use language models to generate vector graphics are usually limited by sequence length, making it difficult to handle complex graphic structures, and the generated results have bottlenecks in semantics and graphic quality.

[0006] Therefore, existing technologies have obvious shortcomings in generating high-quality and well-structured vector graphics, and an efficient and scalable new method is urgently needed to solve the above problems. Summary of the Invention

[0007] In view of this, the present invention provides a method and system for generating vector graphics from text based on a large language model, which can generate detailed and semantically consistent SVG vector graphics.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for generating vector graphics from text based on a large language model, comprising the following steps:

[0010] Collect SVG data and pre-process redundant elements in the SVG data to obtain optimized SVG data;

[0011] The optimized SVG data is rendered to obtain the corresponding rasterized image. The multimodal large model is used to generate corresponding tags for the rasterized image as basic text descriptions, and a multimodal dataset is constructed. Each entry in the multimodal dataset is a triple consisting of the optimized SVG data, the corresponding rasterized image, and the basic text description.

[0012] Input the SVG data and basic text descriptions in the multimodal dataset into the GPT large model to generate a detailed text description for the SVG data;

[0013] The optimized SVG data, the corresponding rasterized images, and detailed text descriptions are combined with the collected instruction dataset as a training set to train the pre-built basic large language model under understanding tasks and generation tasks respectively.

[0014] Furthermore, the method of pre-processing redundant elements in SVG data includes:

[0015] Remove redundant elements from SVG data, including temporary data used by vector editing applications, suboptimal SVG representations, and unused and invisible graphic elements.

[0016] Unify the canvas size and relative coordinate system of each SVG.

[0017] Furthermore, the multimodal large model used is the Tongyi Qianwen multimodal large model; the GPT large model used is ChatGPT or GPT-4.

[0018] Furthermore, during the training phase, for each input SVG data, multiple new SVG semantic tags are added to convert it from the original code form into a structured representation; the new SVG semantic tags added include 15 tag tags, 30 attribute tags and 10 path tags, which replace the original tags and attributes in the SVG source code.

[0019] Furthermore, when adding semantic tags, a description is written for each semantic tag, and each semantic tag is initialized according to the semantic average value of the description text. The initialization process is expressed as:

[0020]

[0021] Among them, s represents a text description; w j represents the jth token of the result after the text description s passes through the word segmenter; n represents the total number of tokens of the result after the text description s passes through the word segmenter; Represents the transpose of the text embedding matrix; E(s) represents the vector obtained by initializing the text description.

[0022] Furthermore, GPT2, Phi-2, or Falcon is selected as the basic large-scale language model.

[0023] Furthermore, when training the basic large language model, for each SVG entry, multiple rounds of dialogue data are sampled from the instruction dataset, and the basic large language model is fine-tuned using the original autoregressive training objective.

[0024] Furthermore, for the understanding task, the SVG data code and the corresponding instruction data are used as input, and the detailed text descriptions generated by the GPT large model are used as training labels to train the basic large language model;

[0025] For the generation task, the instruction data is used as input and the SVG data code is used as the label to train the basic large language model;

[0026] In both comprehension and generation tasks, SVG rasterized images are used as conditional inputs to guide the underlying large language model to understand or generate relevant content.

[0027] Furthermore, the training process of the basic large language model includes two training stages. The first training stage is the feature alignment pre-training stage, in which only the weights of the text embedding matrix are trained; the second training stage is the end-to-end fine-tuning training stage, in which all parameters are fine-tuned in a supervised manner using the entire training set.

[0028] In a second aspect, the present invention provides a system for generating vector graphics from text based on a large language model, comprising:

[0029] Get module, used to get SVG code or text description;

[0030] The inference module is used to generate a description that matches the semantics expressed by the SVG code using the trained large language model described above, or to generate SVG code corresponding to the text description.

[0031] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention can enhance the ability to understand and generate SVG by processing redundant elements in the original SVG code, thereby improving the model's potential representation accuracy. In addition, when describing SVG data, the multimodal large model is first used to generate corresponding labels for the rasterized image as a basic text description, leveraging the advantages of the multimodal large model in image-to-text conversion. The basic text description is then expanded using the GPT large model to obtain a detailed text description, leveraging the advantages of the GPT large model in language generation and context understanding. Through cross-modal (image→text) and text→text relays, efficient and high-precision SVG text annotation generation is achieved, which is more conducive to improving the training accuracy of the basic large model.

[0033] 2. By introducing new semantic tags to replace the tags and attributes of the original SVG code, the present invention can reduce semantic ambiguity and solve the challenge of large language models treating SVG source code as ordinary text, thereby facilitating better understanding and generation of vector graphics that are more in line with human design standards.

[0034] 3. This invention uses SVG data, the corresponding rasterized image, and text description as training data, integrating geometric, visual, and linguistic features to help large language models generate more refined and semantically consistent vector graphics. At the same time, through a two-stage training strategy, it improves the large language model's understanding and performance of SVG generation tasks, laying the foundation for the further development of SVG-related functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 This is a flowchart of the method for generating vector graphics from text based on a large language model provided by the present invention.

[0037] Figure 2 Schematic diagram of the five instruction templates provided by the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, an embodiment of the present invention discloses a method for generating vector graphics from text based on a large language model, comprising the following steps:

[0040] S1. Collect SVG data and pre-process redundant elements in the SVG data to obtain optimized SVG data.

[0041] S2. Render the optimized SVG data to obtain a corresponding rasterized image. Use the multimodal large model to generate corresponding tags for the rasterized image as basic text descriptions, and construct a multimodal dataset; each entry in the multimodal dataset is a triple consisting of the optimized SVG data, the corresponding rasterized image, and the basic text description.

[0042] S3. Input the SVG data and basic text description in the multimodal dataset into the GPT large model to generate a detailed text description for the SVG data;

[0043] S4. Combine the optimized SVG data, the corresponding rasterized image, and the detailed text description with the collected instruction dataset as the training set, and train the pre-built basic large language model under the understanding task and the generation task respectively.

[0044] The above steps are further explained below.

[0045] S1. Collect SVG data and pre-process redundant elements in the SVG data to obtain optimized SVG data.

[0046] SVG data collected from the internet often contains noise, and using it directly for learning may reduce the model's potential representation accuracy. About half of the data in an SVG file is redundant for visual rendering results. This redundancy includes: (1) temporary data used by vector editing applications; (2) non-optimal structural representation of SVG; and (3) unused and invisible graphic elements. Therefore, in order to ensure the accuracy of the model representation, it is necessary to remove the redundant elements in the SVG data.

[0047] Furthermore, the canvas size and relative coordinate system of each SVG must be unified. Specifically, the canvas size can be adjusted to 128×128, and the coordinates are rounded to two decimal places. This preprocessing of SVG data aims to losslessly reduce the size of SVG files generated by vector editing applications.

[0048] S2. After optimizing the SVG, we render it into a 512×512 pixel image using the cairosvg library. The Tongyi Qianwen multimodal model then generates corresponding tags as basic text descriptions. This results in a multimodal dataset where each entry is a triple consisting of an optimized SVG, the corresponding rasterized image, and a basic text description generated by the Tongyi Qianwen multimodal model.

[0049] In a specific embodiment, approximately 250,000 colorful and complex vector graphics were manually collected and normalized according to S1-S2 to ensure that the collected data conforms to a unified standard, including a unified canvas size, relative coordinate system, and representation method.

[0050] S3. Re-annotating SVGs: Based on the rendering results in S2, we used the Tongyi Qianwen multimodal large model to label the corresponding text descriptions. These descriptions are sufficient as textual hints, but too short to be used for SVG annotation. Inspired by the recent success of GPT models in text annotation tasks, we used ChatGPT / GPT-4 to generate instruction fine-tuning data as detailed text descriptions for SVGs. Ultimately, we collected a dataset containing 250,000 annotated, high-quality, and standardized vector graphics, as well as 580,000 unique SVG-text-image samples.

[0051] S4. Model training.

[0052] 1) During the training phase, for each input SVG data (i.e. SVG code X v ), adding multiple new SVG semantic tags in the tokenizer to convert it from the original code form to a structured representation.

[0053] To this end, the present invention defines 55 SVG semantic tags, including 15 tag tags, 30 attribute tags and 10 path tags, and replaces the original tags and attributes in the SVG source code with these new semantic tags, thereby avoiding encoding SVG tags and attributes as ordinary text. <path>will be tokenized as an SVG semantic tag, rather than being processed as a literal "path" by the tokenizer. This ensures the uniqueness of SVG tags and attributes, and allows them to be efficiently integrated into large language models in a manner consistent with the SVG definition, while optimizing token initialization.

[0054] The present invention adds the aforementioned 55 SVG semantic tags to the tokenizer and learns the embedding of these new tags by training the text embedding matrix. When adding semantic tags, a description is written for each semantic tag, and each semantic tag is initialized based on the semantic average of the description text. The initialization process is expressed as follows:

[0055]

[0056] Among them, s represents a text description; w j represents the jth token of the result after the text description s passes through the word segmenter; n represents the total number of tokens of the result after the text description s passes through the word segmenter; Represents the transpose of the text embedding matrix; E(s) represents the vector obtained by initializing the text description.

[0057] This initialization provides a good starting point for each SVG tag and builds a compact distributed representation for all SVG tags.

[0058] 2) Selection of basic large language model architecture.

[0059] We choose GPT2, Phi-2, or Falcon as the base large language models because they can understand both visual and textual data and, among existing open-source models, have demonstrated effective instruction following properties on a variety of language tasks. In theory, other large language models with similar capabilities could also serve as the basis for our approach.

[0060] 3) Training strategy.

[0061] When training the basic large language model, for each SVG entry, multiple rounds of dialogue data are sampled from the instruction dataset, and the basic large language model is fine-tuned using the original autoregressive training objective.

[0062] For the understanding task, convert the SVG code to v and the corresponding instruction data X inst As input, a detailed text description X generated by the GPT-4 large model is used a The large language model is trained as a training label so that the large language model outputs a text description of the SVG code.

[0063] For the generation task, the instruction data is used as input, the SVG data code is used as the label, and the SVG code X v Part of it will be hidden to train the basic language model, SVG code X v The target that the model needs to predict.

[0064] In both comprehension and generation tasks, SVG rasterized images are used as conditional inputs to guide the underlying large language model to understand or generate relevant content.

[0065] like Figure 2 As shown in Figure 2, the present invention develops five different instruction templates specifically for vector graphics generation and understanding tasks. Specifically, types #1 and #2 are helpful for generation tasks, while types #3, #4, and #5 focus on understanding tasks. {prompt} represents a short image description generated by BLIP, {n_paths} i Represents the total number of primitives in group i, {desc} provides a detailed description generated by GPT-4, and Token represents different types of SVG semantic tags. i This represents a complete set of SVG primitives, including a structured set of SVG semantic tags essential for fully describing vector graphics. Types #1-#5 provide a structured framework for training SVG semantic tags, facilitating more accurate vector representation and understanding. Loss is calculated solely based on model responses. This tag represents the end of a sentence.

[0066] "SYSTEM" is a command that describes the task type, specifically the context of the conversation. "ASSISTANT" represents the output generated in response to the command, representing the reply of the large language model. "USER" refers to the input data provided by the user.

[0067] Next, we will further explain the two stages of model training.

[0068] The training process of the basic large language model includes two training phases:

[0069] The first training phase is feature alignment pre-training. In this phase, these pairs are converted into instruction fine-tuning data using the simple extension method described above. Each example can be regarded as a single turn of dialogue. The weights of the visual encoder and the large language model are frozen, and only the weights of the text embedding matrix are trained.

[0070] The second training stage is the end-to-end fine-tuning training stage, in which all parameters are fine-tuned in a supervised manner using the entire training set.

[0071] The present invention considers two specific use case scenarios:

[0072] Efficient parameter fine-tuning: Methods like LoRA fine-tune only a small number of additional model parameters, significantly reducing computational and storage costs while providing performance comparable to fully fine-tuned models. This approach facilitates training and storing large language models on consumer-grade hardware.

[0073] Fully fine-tuning: Fine-tuning all parameters requires higher computational resources, especially in large language models.

[0074] In the inference phase of the model, for the understanding task, given an SVG code, the model will generate a description that matches the semantics expressed by the SVG. For the generation task, the model will generate an SVG based on the input text prompt. In the training and inference phases, the SVG rendered image X img Both can be used as conditional inputs of the model to guide the model to understand or generate relevant content.

[0075] In general, the method of the present invention achieves the highest performance among SVG generation methods based on large language models (LLMs), and only requires model inference. Compared with optimization-based vector graphics generation methods, that is, generation through step-by-step iteration, there is no need for a time-consuming optimization process, and the SVG generation time is significantly shortened.

[0076] In addition, due to the limited support of most large language models for continuous numerical data, the SVGs generated by these models often exhibit imprecise coordinate and color representations, often relying on integers instead of decimals, and using basic color names (such as black or blue) instead of more precise hexadecimal codes. The method of the present invention addresses these shortcomings by converting multiple consecutive single-digit numbers in the coordinates into a mark and using hexadecimal color codes instead of basic color names, thereby expanding the model's ability to represent a wider and more accurate range of colors. The method utilizes the power of LLM to understand and generate SVG, enabling LLM to directly interpret SVG source code and generate high-quality SVG. These generated SVGs exhibit a certain degree of complexity and are consistent with human design principles. The present invention also introduces a structured SVG encoding method to solve the challenge of LLM treating SVG source code as ordinary text.

[0077] In another embodiment, the present invention provides a system for generating vector graphics from text based on a large language model, comprising:

[0078] Get module, used to get SVG code or text description;

[0079] The inference module is used to generate a description that matches the semantics expressed by the SVG code using the trained large language model described above, or to generate SVG code corresponding to the text description.

[0080] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0081] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.< / path>

Claims

1. A method for generating vector graphics from text based on a large language model, characterized in that: The following steps are involved: Collect SVG data and pre-process redundant elements in the SVG data to obtain optimized SVG data; The optimized SVG data is rendered to obtain the corresponding rasterized image. The multimodal large model is used to generate corresponding tags for the rasterized image as basic text descriptions, and a multimodal dataset is constructed. Each entry in the multimodal dataset is a triple consisting of the optimized SVG data, the corresponding rasterized image, and the basic text description. Input the SVG data and basic text descriptions in the multimodal dataset into the GPT large model to generate a detailed text description for the SVG data; The optimized SVG data, the corresponding rasterized images, and detailed text descriptions are combined with the collected instruction dataset as a training set to train the pre-built basic large language model under understanding tasks and generation tasks respectively.

2. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: Methods for preprocessing redundant elements in SVG data include: Remove redundant elements from SVG data, including temporary data used by vector editing applications, suboptimal SVG representations, and unused and invisible graphic elements. Unify the canvas size and relative coordinate system of each SVG.

3. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: The multimodal large model used is the Tongyi Qianwen multimodal large model; the GPT large model used is ChatGPT or GPT-4.

4. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: During the training phase, for each input SVG data, multiple new SVG semantic tags are added to convert it from the original code form into a structured representation; the new SVG semantic tags added include 15 tag tags, 30 attribute tags and 10 path tags, which replace the original tags and attributes in the SVG source code.

5. The method for generating vector graphics from text based on a large language model according to claim 4, characterized in that: When adding semantic tags, write a description for each semantic tag and initialize each semantic tag according to the semantic average value of the description text. The initialization process is expressed as: Among them, s represents a text description; w j represents the jth token of the result after the text description s passes through the word segmenter; n represents the total number of tokens of the result after the text description s passes through the word segmenter; Represents the transpose of the text embedding matrix; E(s) represents the vector obtained by initializing the text description.

6. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: Choose GPT2, Phi-2, or Falcon as the base large language model.

7. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: When training the basic large language model, for each SVG entry, multiple rounds of dialogue data are sampled from the instruction dataset, and the basic large language model is fine-tuned using the original autoregressive training objective.

8. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: For the understanding task, the SVG data code and the corresponding instruction data are used as input, and the detailed text description generated by the GPT large model is used as the training label to train the basic large language model; For the generation task, the instruction data is used as input and the SVG data code is used as the label to train the basic large language model; In both comprehension and generation tasks, SVG rasterized images are used as conditional inputs to guide the underlying large language model to understand or generate relevant content.

9. The method for generating vector graphics from text based on a large language model according to claim 1, characterized in that: The training process of the basic large language model consists of two training phases. The first training phase is the feature alignment pre-training phase, in which only the weights of the text embedding matrix are trained. The second training stage is the end-to-end fine-tuning training stage, in which all parameters are fine-tuned in a supervised manner using the entire training set.

10. A system for generating vector graphics from text based on a large language model, characterized in that: include: Get module, used to get SVG code or text description; An inference module is configured to generate a description that is consistent with the semantics expressed by the SVG code, or to generate SVG code corresponding to the text description, using the trained large language model according to any one of claims 1 to 9.

Images