Graphical design method and device, electronic equipment, storage medium, program product

Abstract

The present disclosure provides a graphic design method and device, electronic equipment, storage medium and program product, and belongs to the technical field of artificial intelligence. The method comprises the following steps: interacting with an object through a multi-modal large language model to obtain design intention information of the object, and converting the design intention information into design element information; mapping the design element information to a same multi-modal feature space through a pre-trained visual language model to obtain a multi-modal feature sequence of the design element information; processing the multi-modal feature sequence through a transparent graph generation model to generate a transparent base graph of a design element; and splicing and integrating the transparent base graph of the design element to generate a design graph.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a graphic design method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Graphic design is a broad field encompassing visual arts and communication that creates visual representations of information and ideas through visual elements and design principles. These elements include typography, logos, illustrations, photographs, and symbols, while design principles cover aspects such as layout, contrast, balance, repetition, alignment, color theory, and space utilization. Summary of the Invention

[0003] This disclosure provides a graphic design method, which includes: interacting with an object through a multimodal large language model to obtain design intent information of the object, and converting the design intent information into design element information; mapping the design element information to the same multimodal feature space through a pre-trained visual language model to obtain a multimodal feature sequence of the design element information; processing the multimodal feature sequence through a transparency image generation model to generate a transparent background image of the design elements; and stitching and integrating the transparent background images of the design elements to generate a design drawing.

[0004] This disclosure provides a graphic design apparatus, comprising: an interaction unit for interacting with an object through a multimodal large language model to obtain design intent information of the object and convert the design intent information into design element information; a mapping unit for mapping the design element information to the same multimodal feature space through a pre-trained visual language model to obtain a multimodal feature sequence of the design element information; a processing unit for processing the multimodal feature sequence through a transparency image generation model to generate a transparent background image of the design elements; and a generation unit for stitching and integrating the transparent background images of the design elements to generate a design drawing.

[0005] This disclosure provides an electronic device, including: one or more processors; and a memory configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the method described in any embodiment of this disclosure.

[0006] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to execute the methods described in any embodiment of this disclosure.

[0007] This disclosure provides a computer program product, including a computer program that, when executed by a computer, implements the methods described in any embodiment of this disclosure. Attached Figure Description

[0008] Figure 1 This is a flowchart of a graphical design method according to an exemplary embodiment of the present disclosure.

[0009] Figure 2 This is a schematic diagram of a graphical design autonomous intelligent agent for general artificial intelligence, as described in an exemplary embodiment of this disclosure.

[0010] Figure 3 This is a schematic diagram illustrating multimodal communication between a user and an intelligent agent, according to an exemplary embodiment of this disclosure.

[0011] Figure 4 This is a schematic diagram illustrating the fine-tuning of a multimodal pre-trained model according to an exemplary embodiment of the present disclosure.

[0012] Figure 5 This is a schematic diagram illustrating the generation of a transparent background map of various design elements according to an exemplary embodiment of this disclosure.

[0013] Figure 6 This is a schematic diagram of a design drawing generated according to an exemplary embodiment of this disclosure.

[0014] Figure 7 This is a schematic diagram of the structure of a graphic design apparatus according to an exemplary embodiment of the present disclosure.

[0015] Figure 8 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of this disclosure.

[0017] Figure 1 This is a flowchart of a graphical design method according to an exemplary embodiment of the present disclosure. Figure 1 The methods provided in the embodiments can be executed by any electronic device, such as a terminal and / or a server, and this disclosure does not limit them. Figure 1 As shown, the method provided in this embodiment includes the following steps.

[0018] In S110, the design intent information of the object is obtained by interacting with the object through a multimodal large language model, and the design intent information is converted into design element information.

[0019] The Multimodal Large Language Model (MLLM) in this disclosure is an advanced artificial intelligence model capable of processing and understanding multiple types of information. The MLLM fuses data from different modalities (e.g., any two or more from images, audio, video, and text) and trains a unified model to achieve cross-modal understanding and generation. The MLLM can employ a Transformer-based architecture and incorporate a multimodal encoder to process data from different modalities. During training, the MLLM learns the correlations and complementarities between various modalities, thereby achieving cross-modal semantic understanding and generation. The MLLM in this disclosure enables artificial intelligence to possess natural language understanding and generation, image recognition, and interactive capabilities for speech and video.

[0020] In some embodiments, the multimodal large language model in this disclosure can develop a unified user interface at the system layer, possessing the ability to input and output multiple modal data. Implementation-wise, multimodal capabilities can be implemented on the terminal by calling models of different modalities or even APIs (Application Programming Interfaces). In other embodiments, language models, visual models, and sound models are connected at the technical framework layer. These models learn independently, are trained using data from different modalities, and then the concatenated model is further pre-trained on cross-modal data and fine-tuned on different task data to generate the multimodal large language model. In still other embodiments, from the training phase, the multimodal large language model utilizes a large amount of data from different modalities for pre-training, achieving tight coupling technically. It not only achieves multimodality at the input and output ends but also possesses powerful multimodal reasoning and cross-modal transfer capabilities.

[0021] In multimodal data processing, the Transformer architecture can capture the complex relationships between different modalities and generate joint feature representations. Multimodal encoders are responsible for converting inputs from different modalities into feature representations that the model can understand. These encoders can be specifically designed for different modalities, such as video encoders, image encoders, table encoders, and text encoders. Video encoders decompose video data into keyframes and motion components, then convert them into feature representations. These feature representations can contain both spatial content and temporal information from the video. Image encoders convert image data into feature representations, involving feature extraction—that is, extracting useful information from images, such as color, texture, shape, and object location. Image encoders include NFNet, ViT, CLIP ViT, etc. Table encoders convert tabular data into feature representations. One approach is to convert the tabular data into text format and then process it using a text encoder; another approach is to use encoders specifically designed for tabular data, such as Transformer-based models to process rows and columns in tables. Text encoders convert text data into feature representations, involving steps such as text segmentation, embedding, and encoding. Text encoders include pre-trained language models such as BERT and GPT. Encoded data from different modalities are fused together for joint representation. This fusion can utilize attention mechanisms, adaptively fusing information from different modalities by calculating mutual attention weights between them. This allows the model to better focus on important modal information.

[0022] In practical implementation, deep learning frameworks (such as PyTorch or TensorFlow) can be used to build a Transformer-based multimodal large language model. First, the data from different modalities is preprocessed, such as through word segmentation and image preprocessing, to convert it into an input format acceptable to the model. An independent encoder is built for each modality, and the outputs of these encoders are used as input to the Transformer model. The Transformer architecture is then used to process the multimodal data. This involves multiple Transformer layers, each containing a multi-head self-attention mechanism and a feedforward neural network. At the output of the Transformer model, a fusion layer can be used to integrate information from different modalities. This can be achieved by calculating the mutual attention weights between modalities. Depending on the specific task (such as classification, generation, detection, etc.), the output layer is designed to generate the model's final output.

[0023] The object in this disclosure embodiment can be any user performing graphic design using the methods provided in this disclosure embodiment. The user can interact with the multimodal large language model through a client installed on their terminal or an open webpage. Through this interaction, the multimodal large language model obtains the user's design intent information, i.e., any relevant information related to the design drawing the user wishes to generate. In graphic design, design intent information refers to the intention or goal that the user (e.g., a designer) hopes to express through the design drawing during the design process; it determines the direction and style of the design drawing.

[0024] In this embodiment, the multimodal large language model can understand the design intent information input by the user, thereby transforming the abstract design intent information into specific design element information. Design element information refers to any information related to a design element. A design element is a basic component or unit that constitutes a design drawing and is a tool used to express design intent and create visual effects. Design elements include, but are not limited to, one or more of the following: lines, shapes, colors, textures, fonts, spaces, etc.

[0025] In an exemplary embodiment, interacting with an object through a multimodal large language model to obtain the object's design intent information and converting the design intent information into design element information includes: receiving first design intent information sent by the object through the multimodal large language model and converting the first design intent information into first design element information; filling the first design element information into the corresponding position of a graphic design element template; detecting that there are unfilled design elements in the graphic design element template and triggering the multimodal large language model to send query information corresponding to the unfilled design elements to the object; receiving second design intent information returned by the object in response to the query information through the multimodal large language model and converting the second design intent information into second design element information.

[0026] The design intent information includes the first design intent information and the second design intent information, and the design element information includes the first design element information and the second design element information.

[0027] In this embodiment of the disclosure, a pre-defined graphic design element template can be provided. This template includes the design elements that need to be obtained from the user for a given design drawing. Instructions can be added to the multimodal large language model to instruct it to obtain design element information corresponding to all design elements contained in the graphic design element template through multiple rounds of interaction with the user. It is understood that "design element information corresponding to all design elements" here includes design element information that the user explicitly indicates cannot be provided or will not be provided.

[0028] For example, a multimodal large language model can interact with the user in multiple rounds. After understanding the user's initial design intent, it converts it into initial design element information. This initial design intent can be data entered by the user at any point during the multi-round interaction. The initial design element information is compared with the design elements contained in the graphic design element template, and the information is filled into the corresponding positions. Then, guided by instructions, the multimodal large language model checks if there are any unfilled design elements in the graphic design element template (referred to as unfilled design elements, i.e., design elements whose corresponding design element information has not yet been filled). If so, the multimodal large language model, guided by instructions, generates a query for the unfilled design element and returns the query to the user to guide the user in answering it, thereby continuing to obtain design element information related to the unfilled design element from the user; this is called second design element information. This process is repeated until the multimodal large language model obtains design element information corresponding to all design elements in the graphic design element template.

[0029] The multimodal large language model provided in this embodiment can gradually guide the user to provide complete design element information, making the vague design intent gradually clear, without requiring the user to provide complete design intent information at once, thus reducing the requirements on the user, while ensuring the integrity of the obtained design element information.

[0030] In an exemplary embodiment, converting the second design intent information into second design element information includes: if the second design intent information indicates that the unfilled design element is empty, generating second design element information corresponding to the unfilled design element based on the first design element information; sending the second design element information to the object through the multimodal large language model; and determining the second design element information in response to the object's feedback information on the second design element information.

[0031] In this embodiment, the user is allowed to return a second design intent information that is not empty, meaning the user does not provide design intent information corresponding to one or more unfilled design elements. In this case, the multimodal large language model can understand the second design intent information and fill in nil (representing emptiness) in the design element corresponding to the graphic design element template. In subsequent steps, based on the design element information already obtained from the user (including the aforementioned first design element information), the second design element information corresponding to the unfilled design elements can be automatically generated. A corresponding transparent background image is then generated based on this automatically generated second design element information. This transparent background image is then displayed to the user through the multimodal large language model. If the user's feedback indicates acceptance of the automatically generated transparent background image, the second design element information can be filled into the position of the design element corresponding to the graphic design element template. If the user's feedback indicates rejection of the automatically generated transparent background image, new and different second design element information and its corresponding transparent background image can be generated and displayed to the user again, until the user's acceptance feedback is received.

[0032] The method provided in this disclosure can automatically supplement design elements not provided by the user, and the automatically supplemented design elements are confirmed by the user through the interaction of a multimodal large language model, which can both ensure the integrity of the obtained design element information and meet the user's design needs.

[0033] In an exemplary embodiment, obtaining the design intent information of an object through interaction with a multimodal large language model includes: interacting with the object through the multimodal large language model to obtain preliminary design intent information of the object; converting the preliminary design intent information into preliminary design element information; retrieving a matching target subgraph from a knowledge graph of the graphic design domain based on the preliminary design element information; generating preliminary adjustment information for the preliminary design intent information based on the target subgraph; and obtaining the design intent information in response to feedback information from the object regarding the preliminary adjustment information.

[0034] In this embodiment of the disclosure, a large number of recognized and aesthetically pleasing sample design graphs can be collected. These sample design graphs are then used to train a GNN (Graph Neural Network), which generates a domain knowledge graph (which can be represented as a Domain Knowledge Graph). This domain knowledge graph includes multiple recognized, aesthetically pleasing subgraphs that satisfy the design principles of the domain.

[0035] In this embodiment of the disclosure, during the interaction with the user through a multimodal large language model, the user's preliminary design intent information is obtained. This information is then understood through the multimodal large language model and converted into preliminary design element information. Based on this preliminary design element information, the subgraph with the highest similarity to the generated graphic design domain knowledge graph is selected as the target subgraph for matching. The similarity between the preliminary design element information and the target subgraph determines whether the preliminary design drawing generated from the preliminary design element information meets the aesthetic standards of the graphic design domain, i.e., whether the preliminary design drawing is aesthetically pleasing. For example, if the similarity is greater than a preset threshold, the preliminary design drawing is considered aesthetically pleasing and does not require adjustment; it can be used as the final design drawing. If the similarity is less than or equal to the preset threshold, the preliminary design drawing is considered not aesthetically pleasing. Furthermore, by comparing it with the target subgraph, discordant design elements in the preliminary design drawing are identified, modification suggestions are provided, and preliminary adjustment information for the preliminary design intent information is generated based on these suggestions. The initial adjustment information is fed back to the user through a multimodal large language model. If the user's feedback indicates acceptance of the initial adjustment information, the initial design intent information is adjusted accordingly to obtain the final design intent information. If the user's feedback indicates rejection of the initial adjustment information, the initial design intent information can continue to be used as the final design intent information.

[0036] In this embodiment of the disclosure, the preliminary design intent information provided by the user can be automatically adjusted based on the knowledge graph of the graphic design field to make it more in line with the aesthetic design principles of the graphic design field.

[0037] In this embodiment, user intent recognition and understanding can be achieved through a multimodal large language model. By engaging in dialogue with the user, the system understands their design requirements. Machine vision technology is used to refine the user's design intent into specific design elements. Specifically, a multimodal large language model based on the Transformer architecture can be used, capable of processing data in multiple modalities such as text and images. Through dialogue with the user, the multimodal large language model can capture the user's input in multiple modalities, including natural language and images, and understand their design requirements. The multimodal large language model can continuously interact with the user, gradually clarifying and refining the user's design intent through contextual information. For user input images, the multimodal large language model can understand the user's intent using machine vision technology. The multimodal large language model incorporates basic machine vision capabilities such as segmentation, detection, recognition, and Optical Character Recognition (OCR) to assist in extracting design elements, such as text, shapes, and colors, from the user-provided images. By combining with the multimodal large language model, the system can match and refine the user's design intent with the information extracted by machine vision.

[0038] In an exemplary embodiment, the converted design element information can be integrated into a structured format file, such as JSON (JavaScript Object Notation) or XML (Extensible Markup Language). JSON is used as an example below, but this disclosure is not limited to this. These files can contain detailed information about design elements, such as text style, font, font size, typeface, as well as visual elements and layout planning information.

[0039] In S120, the design element information is mapped to the same multimodal feature space by a pre-trained visual language model to obtain the multimodal feature sequence of the design element information.

[0040] In this embodiment of the disclosure, the pre-trained visual-language model refers to a pre-trained model that simultaneously models and represents image and language (text / image) data, and is applied to multiple tasks such as vision, language, and cross-modal processing. A multimodal feature space refers to a system where data from different modalities (such as text, images, audio, and video) are transformed and represented into a unified feature space for subsequent computation, analysis, and fusion.

[0041] In this embodiment of the disclosure, based on the design element information identified in step S110, a pre-trained visual language model is used to map the multimodal text, images, etc. contained in the design element information to the same feature space in order to generate a multimodal feature sequence. Here, the multimodal feature sequence refers to the sequence formed by organizing features of different modalities in a certain order when processing multimodal data.

[0042] In this embodiment of the disclosure, the multimodal large language model supports user input of multimodal data, such as text and images. The address of the user-input image is stored in the structured format file output by the multimodal large language model. Then, a pre-trained visual language model (such as ViLBERT, UNITER, etc.) maps the text and input image contained in this structured format file to the same multimodal feature space. The pre-trained visual language model has learned how to map text and images to the same multimodal feature space through pre-training. These models are able to capture the semantic relationships between text and images.

[0043] This disclosure enables multimodal interaction with users through a multimodal large language model, allowing users to interact with the system using different modalities (such as voice, text, and images). A pre-trained visual language model can generate corresponding outputs based on the different modalities of user input.

[0044] For example, a pre-trained visual language model uses an appropriate feature extraction method (such as a Convolutional Neural Network (CNN) for the input image, and a Recurrent Neural Network (RNN) or Transformer for the text) to extract its low-level features for each modality. The extracted features are mapped to a shared embedding space. This space is a high-dimensional vector space where feature vectors from different modalities can be compared using some metric (such as cosine similarity). Alignment techniques (such as contrastive learning and canonical correlation analysis) are used in the shared embedding space to ensure that feature vectors from different modalities reflect the same semantic information. Fusion techniques (such as feature concatenation, feature addition, and feature multiplication) can be used to further combine feature vectors from different modalities to generate richer representations.

[0045] In S130, the multimodal feature sequence is processed by a transparency map generation model to generate a transparent base map of the design elements.

[0046] In this embodiment, the transparent image generation model refers to a machine learning model capable of converting design elements corresponding to a multimodal feature sequence into a transparent base map containing that design element. In this embodiment, the transparent image generation model generates a separate transparent base map for each design element, meaning one design element corresponds to one layer. This allows users to modify the corresponding transparent base map when they wish to modify a design element, without affecting other design elements.

[0047] In this embodiment of the disclosure, a transparent background image refers to an image in which the background part of the graphic is set to transparent, and the background part has no color or texture. This allows the transparent background images of various design elements to be freely combined without being restricted by the background. They can be easily superimposed and synthesized with other design elements, thereby creating more visual effects and creative expressions.

[0048] In S140, the transparent background images of the design elements are stitched together and integrated to generate a design drawing.

[0049] In some embodiments, during user interaction with a multimodal large language model, the obtained design intent information may include a user-inputted reference layout image, which the user indicates is intended as a layout reference for the design drawing. The system can then stitch together the transparent background images of various design elements based on this reference layout image to generate the design drawing.

[0050] In an exemplary embodiment, stitching together the transparent background images of the design elements to generate a design drawing includes: adjusting the transparent background images of the design elements according to a knowledge graph in the graphic design domain; and generating the design drawing based on the adjusted transparent background images of the design elements.

[0051] In other embodiments, design elements that do not conform to graphic design principles or are aesthetically unappealing can be detected in the preliminary design drawing after the transparent background of the design element is stitched together, based on the knowledge graph of the graphic design domain. Modification suggestions can then be proposed. If the user accepts the modification suggestions, the design drawing can be regenerated based on the transparent background of the adjusted design element.

[0052] In an exemplary embodiment, the method provided by this disclosure further includes: generating explanatory information for the design drawing based on the graphic design domain knowledge graph, the preliminary adjustment information, and the feedback information.

[0053] In this embodiment of the disclosure, based on the knowledge graph of the graphic design domain and the preliminary adjustment information and feedback information obtained during the interaction with the user through the multimodal large language model, explanatory information of the design drawing can be generated, such as why it is designed this way and what the reasons are for this layout, thereby helping the user to better understand the design drawing.

[0054] The graphic design method provided in this disclosure, by combining a multimodal large language model, a pre-trained visual language model, and a transparent image generation model, achieves accurate understanding and implementation of the user's design intent. This intelligent design assistance system can greatly improve design efficiency and accuracy, providing designers with more convenient and efficient design tools. Furthermore, by generating a separate transparent background image for each design element, it is convenient to make individual modifications to each design element without affecting other design elements.

[0055] This disclosure provides an autonomous intelligent agent for graphic design, moving towards general artificial intelligence. This agent can be used to implement the graphic design method provided in this disclosure, generating multi-layered, easily editable, and interpretable graphic designs (i.e., design diagrams). Multi-layered design means that each design element is on a separate layer, resulting in a hierarchical design. Since each design element corresponds to a separate layer, each design element can be edited individually, and explanations can be provided for each individual design element. Ease of editing is primarily due to the layered nature of the design elements; changing one design element will not affect others. Another improvement is in the method itself. While mainstream AI algorithms in related technologies focus on decoupling features in the feature space, this disclosure provides a more direct approach. Interpretability arises because this disclosure involves a multi-turn dialogue between the user and a multimodal large language model, requiring higher-level artificial intelligence. In other words, the multimodal large language model itself knows which design elements exist, making it easier for it to understand the design intent, provide reasonable and correct explanations, and generate detailed explanatory information. Interpretability also stems from the model's higher level of intelligence and understanding of knowledge.

[0056] First, a graphical design autonomous intelligent agent for general artificial intelligence, proposed according to an embodiment of the present disclosure, will be described with reference to the accompanying drawings. Figure 2 A flowchart illustrating the graphical design of autonomous intelligent agents toward general artificial intelligence according to one embodiment of this disclosure.

[0057] like Figure 2 As shown, Figure 2 The process involves four stages: Phase 1: constructing a multimodal graphic design dataset; Phase 2: interpreting multimodal inputs; Phase 3: aesthetic collaborative planning; and Phase 4: accurate multi-layered generation.

[0058] Specifically, the first phase involves constructing a multimodal graphics design database as a multimodal large language model (i.e., MLLM 220) and a pre-trained visual language model. Figure 2 (not shown in the image), transparency map generation model (i.e.) Figure 2 The training dataset for the visual generator (260) is used to train the entire model end-to-end, including MLLM 220, a pre-trained visual language model, and a transparent graph generation model. The multimodal graphics design database includes multimodal input data samples and corresponding design graph samples as labels.

[0059] The second stage refers to the MLLM 220 performing intent recognition or interpretation on the user's multimodal input data to obtain a structured document, namely structured intent 230. This structured intent 230 includes design element information converted from design intent information. Specifically, the user can interact with the MLLM 220 through interactive multimodal input 210 to obtain the user's design intent information. This interactive multimodal input 210 can include text 211, images 212, and interactive features 213. Interactive features 213 refer to the MLLM 220's ability to ask questions based on the user's input data (e.g., the first design intent information), allowing the user to provide further design intent information (e.g., the second design intent information) in response to these questions. Figure 2 It is assumed that the structured intent 230 is in JSON format.

[0060] In the third stage, collected graphic design works (e.g., the multimodal graphic design database built in the first stage) can be annotated at the granularity of design elements to learn the aesthetic graphs in the graphic design works, thereby generating a domain knowledge graph 240. When receiving design element information contained in the user's structured intent 230, the corresponding sub-graph 241 can be found in the learned graphic design domain knowledge graph 240 as the target sub-graph. The similarity between sub-graph 241 and the design element information in the structured intent 230 is calculated. Suggestions are then given to the user. For example, the similarity of sub-graph 241 can be calculated using an LLM (Large Language Model) 250 to identify which design element information is recommended for modification, generating preliminary adjustment information. This preliminary adjustment information can include suggestions for which design elements(s) to modify and why, and is fed back to the user. If the user accepts, the adjusted intent 230' is obtained as the design element information corresponding to the design intent information. The adjusted intent 230' is still in JSON format. Then, the adjusted intent 230' is input into the visual generator 260 for processing, resulting in multi-layered generated designs 270, where the generated design is composed of multiple layers of transparent underlying design elements. For example, the LLM 250 can also generate explanations for the given suggestions 280, which serve as explanatory information for the design.

[0061] This disclosure provides an autonomous intelligent agent for graphic design moving towards general artificial intelligence, which generates multi-layered, easily editable, and interpretable graphic designs, including the following steps and modules.

[0062] Step 1. User intent identification and understanding.

[0063] Specifically, based on a multimodal large language model, it engages in dialogue with users and continuously interacts to understand the user's rough design intent information. Through basic machine vision capabilities such as segmentation, detection, recognition, and optical character recognition, it refines the design intent information down to each design element, including text style, font, font size, glyph, as well as visual elements and layout planning information, and outputs it as a structured format file, such as JSON.

[0064] Visual elements are the fundamental components of a visual object or design. Examples of visual elements include: lines, shapes, colors, textures, space, volume, and light and shadow. Layout planning in graphic design primarily refers to the organization, arrangement, and configuration of graphic elements in two-dimensional space during the visual design process. The aim is to ensure a harmonious overall visual effect, clear information delivery, and to guide the viewer's eye along the designer's intended path. Key elements of layout planning include, but are not limited to: balance and symmetry, alignment and consistency, hierarchy and depth, and color and contrast.

[0065] A user's design intent might be a very vague and abstract description, possibly accompanied by some reference images. For example, a user might want to design an apple poster. The user might tell the AI ​​(including a multimodal large language model), "I want to design an apple poster, in a cute style. An apple looks something like this," and then provide a reference image. The AI ​​can understand both text and image input. When understanding image input, it can utilize basic machine vision capabilities such as segmentation, detection, recognition, and OCR. Once the AI ​​receives this information, due to the specific nature of graphic design, it needs to know what the text characters are and what the non-text decorations are. The AI ​​can then use the recognized information to match the corresponding design elements. For example, in the apple poster example, the text is "apple," and the image is the apple picture provided by the user. Using this information as a condition, it generates a multi-layered transparent background image of design elements, and then combines these multi-layered transparent background images.

[0066] In this embodiment of the disclosure, the description of abstract design intent information is broken down into the details of each design element. For example, what color or font the user wants. If the user doesn't know, leave it blank, and then generate a single design element or a transparent background image of a single design element based on the information obtained from the user to display to the user. If the user likes it, keep it; if not, regenerate it.

[0067] In this embodiment, a multimodal large language model (MLM) interacts with the user, autonomously and continuously interacting through pre-set "Fill-in-the-blank" task instructions to understand the user's rough design intent. That is, the graphic design element template can use blank spaces to indicate the design elements to be filled. For example, what the text in the design drawing is, what the font should be, what the color should be, whether there are any special effects, etc. In other words, all the design element information required for the design drawing is set one by one in the graphic design element template. If the user has already mentioned the corresponding design element information during communication, the corresponding blank is automatically filled. If it hasn't been mentioned, the multimodal large language model asks. If the user doesn't provide it, it fills in "nil". This method guides the multimodal large language model to proactively inquire about the user's ideas and design details, without requiring the user to write a long, detailed description. This embodiment designs a thought chain within the framework of the multimodal large language model, enabling the multimodal large language model to think logically in the context of graphic design, allowing it to autonomously communicate and exchange ideas with the user using a standard format graphic design element template as its target.

[0068] In this embodiment, a multimodal visual language model (i.e., a multimodal large language model), related modules for improving the accuracy of basic visual capabilities (such as basic machine vision capabilities like segmentation, detection, recognition, and optical character recognition), an instruction-guided autonomous task mechanism (i.e., "the multimodal visual language model in the intent recognition and understanding module will set instructions to complete a fill-in-the-blank task, i.e., a preset graphic design element template. Therefore, in this process, the agent will autonomously ask the user to complete the preset task." This means giving the multimodal large language model some instructions so that the model knows what to do), and a structured output format are used to achieve user intent recognition and understanding, and to convert it into design element information.

[0069] Specifically, the basic vision tasks required are modularized using a Mixture of Experts (MoE) model, including segmentation, detection, recognition, and optical character recognition. A collaborative mechanism is used to dynamically schedule the necessary sub-models to improve the accuracy of basic vision capabilities. Simultaneously, based on cue learning, a standard output template is provided through a fill-in-the-blank task, requiring the model to use this template as a target, thereby enabling autonomous communication and interaction within the model. Finally, the model is trained using a large amount of paired data, allowing it to ultimately output structured and clearly defined design requirements.

[0070] For example, a graphic design element template and a multimodal large language model are pre-designed. The multimodal large language model is instructed to complete the task of filling the blanks in the graphic design element template through multiple rounds of interaction with the user. For instance, when the multimodal large language model receives the first sentence or image sent by the user, it performs intent understanding on the first sentence or image, and then fills the recognized content into the corresponding blanks in the graphic design element template. If it finds that there are still blanks in the template, the model asks the user questions about the other blanks, receives the second sentence or image sent by the user, and repeats this process until all blanks in the template are filled.

[0071] For example, this multimodal large language model incorporates multiple sub-models, including segmentation, detection, recognition, and optical character recognition. When the user inputs an image, the segmentation and detection sub-models are invoked; if the input is speech, the recognition sub-model is invoked; and if the input is text, the optical character recognition sub-model is invoked to extract features from the multimodal input. Then, a self-attention mechanism is used to fuse the features of the multimodal input to understand the user's intent.

[0072] Hybrid expert models are a highly efficient neural network architecture that uses a set of independent sub-models to collaboratively complete a task, dynamically selecting a few experts to perform computations based on the input data. Specially trained expert models are used to process different tasks or data characteristics, aiming to achieve globally optimal results. By organizing multiple expert models in parallel and controlling the output weights of each expert through one or more gating networks, it achieves efficient utilization of computational resources and enhances the model's expressive power.

[0073] Step 2. Based on the information identified in Step 1, use a pre-trained visual language model to map multimodal text and images to the same feature space or a multimodal feature space to generate a multimodal feature sequence.

[0074] This disclosure embodiment utilizes knowledge from the field of graphic design to enhance the pre-trained visual language model with knowledge from the field of graphic design, guiding the pre-trained visual language model to think in a specific direction of graphic design.

[0075] An example, a structured format file, such as JSON, would be used in the apple example above. Assuming the user provides a reference image of an apple, the format file would store the address of that reference image in the non-text elements. When generating related non-text elements subsequently, the input to the pre-trained visual language model is not the plain text "apple," but rather a multimodal input, including both text and images (such as the reference image of the apple).

[0076] In this embodiment of the disclosure, during the stage of understanding the user's intent, multimodal input is accepted, and it is also acceptable for the user not to provide any input. The text input by the user may include descriptive text, and the image input by the user may include optical character images, reference object images (or reference images), style images, etc., all of which can be mapped to the same multimodal feature space through a pre-trained visual language model. For example, some fonts or characters have special designs; if a user wants a certain design or effect but doesn't know how to describe it in words, they can directly send an image as a reference. The descriptive text input by the user can be used to indicate whether the input image is a reference object image, a style image, or an optical character image. A reference object image refers to an image that the agent can use as a reference to generate a similar image. A style image refers to an image that the agent can generate with the same style as the reference image. An optical character image refers to characters input by the user as an image. All images and text are mapped to the same feature space, facilitating subsequent generation.

[0077] Step 3. Fine-tune advanced text-based graph models, such as diffusion-based or flow-based generative models, to obtain a transparent graph generation model. Process the multimodal feature sequence from Step 2 (also known as multimodal pre-trained model features) through this transparent graph generation model to generate transparent base maps for each individual design element.

[0078] In this embodiment of the disclosure, the pre-trained text-based image model can be fine-tuned using feature sequence samples and their corresponding transparent background image samples to obtain a transparent image generation model.

[0079] Step 4. Based on the basic rules of graphic design and aesthetic understanding (including the knowledge graph of the graphic design field mentioned above), stitch together and integrate the transparent background images of the individual design elements generated in Step 3 to generate the final design drawing, and output the design motivation of the complete thought chain as the explanatory information of the final design drawing.

[0080] In an exemplary embodiment, the design intent information input by the user in step 1 includes layout planning information, which may include a reference layout diagram, i.e., the user wishes to design the layout of the design diagram according to the reference layout diagram.

[0081] For example, based on the transparent background image of the individual design elements generated in step 3 and the reference layout image obtained in step 1, an aesthetically perceptive splicing model is used. This involves adjusting and correcting the corresponding design elements or transparent background images to ensure image harmony, based on the agent's own understanding of graphic design rules and aesthetics, and then splicing and integrating them to generate the final design image. Simultaneously, the design motivation of the complete thought chain is output as an explanation of the final design image. For example, this aesthetically perceptive splicing model can include the aforementioned GNN model, meaning its understanding of graphic design rules and aesthetics can include the aforementioned knowledge graph of the graphic design domain. This aesthetically perceptive splicing model can calculate various design attributes based on the position of design elements and the overall style, and identify which design attribute is unreasonable. If so, it proposes to modify the design element with this unreasonable deviation. When considering aesthetics, specific aspects are considered, such as the font, color, position, size, and layout. During training, these design attributes are linked to the judgment of beauty or ugliness, training the aesthetically perceptive splicing model to make judgments based on these design attributes. Therefore, while judging whether something is beautiful or not, we can further pinpoint which design attribute is incorrect. Based on this data, a GNN network can be trained to form a knowledge graph for the graphic design domain based on data-driven aesthetic understanding.

[0082] For example, a user says they want to generate an apple poster. After all the necessary design elements are determined, they need to be combined. This includes the placement and size of each element. This layout information can be gradually optimized during the dialogue with the intelligent agent. During this optimization process, for example, if the user wants to enlarge the font to the center, the aesthetically-aware model, with its basic aesthetic understanding, determines that this would disrupt the balance. Therefore, during the communication between the user and the multimodal language model, it will directly tell the user that this is not good, as it disrupts the balance, and that the font should be smaller and placed in a suitable position. Similarly, when the user selects fonts, colors, etc., the aesthetically-aware model will refer to its own understanding of aesthetics and provide suggestions. Finally, after achieving overall visual balance, the final result is obtained. The process of communication between the user and the multimodal language model can then serve as explanatory information for the design, explaining why it was designed this way, why it was arranged this way, and the specific reasons, covering the complete thought process.

[0083] In this embodiment, an aesthetically perceptive splicing model, based on the rules of graphic design and an understanding of aesthetics, autonomously fine-tunes inharmonious elements to ensure a harmonious final result. Simultaneously, the aesthetically perceptive splicing model (which may include...) Figure 2The LLM 250 in the database uses explanatory information from the database tags to provide an explanation of the intent of the final design drawing that conforms to the thought process.

[0084] Large language models, or multimodal large language models, have opened the door to general-purpose artificial intelligence. By connecting discrete fundamental tasks using natural language, they can create autonomous intelligent agents with the capabilities of understanding, decision-making, planning, and execution—a promising and feasible direction. Natural language understanding, image recognition, and image generation are all fundamental tasks, and LLM (Language Modeling) connects understanding and generation. Currently, there are no autonomous intelligent agents specifically designed for graphic design; therefore, filling this gap is crucial for creative innovation in the era of general artificial intelligence.

[0085] For example, this autonomous agent integrates a multimodal large language model, a pre-trained visual language model, and a transparent graph generation model. The autonomous agent design process for graphical design provided in this disclosure is as follows:

[0086] Input: Human designers (i.e., users) refine design concepts and clarify design requirements through text or images in the interaction, guided by an autonomous intelligent agent. This involves all design elements related to graphic design, including text, images, and typography.

[0087] Output: Graphic design works (i.e. design drawings) designed by the autonomous intelligent agent according to the requirements, as well as explanatory information explaining the design intent of the works;

[0088] 1:while n=1to∞do

[0089] 2: The intent recognition and understanding module performs natural language interaction with user input, allowing users to input text and images. Combined with a multimodal visual language model (i.e., a multimodal large language model) with basic visual understanding capabilities, it will understand and organize all user input.

[0090] 3: The multimodal visual language model in the intent recognition and understanding module will set instructions requiring the completion of a fill-in-the-blank task, i.e., a preset graphic design element template. Therefore, during this process, the agent will autonomously ask the user for help to complete the preset task;

[0091] 4. After obtaining all detailed design intent information from the user, output a structured design list, transforming the user's rough design concept into information for each design element. i Where i is a positive integer greater than or equal to 1;

[0092] 5: Based on elementsi Call the fine-tuned pre-trained visual language model, and add elements i Mapped to the same feature space ε(·);

[0093] 6: Then the feature sequence of all design element information can be uniformly represented as f i =ε(elements) i );

[0094] 7: Utilizing advanced generative model frameworks, such as diffusion-based or flow-based generative models, a transparent graph generation model D(·) is obtained. D(·) will generate multimodal feature sequences (i.e., f... i =ε(elements) i The sequence of elements is used to generate a visual representation of a transparent background. i ;

[0095] 8: Visualize individual visual expressions i Input assembly module (e.g., aesthetically pleasing assembly model) do

[0096] 9: Aesthetic Perception Model Analysis: Visual Expression of All Design Elements i Based on an understanding of graphic design rules and aesthetics, adjustments and modifications are made to the corresponding design elements to obtain the adjusted visual expression. And visual expression that does not require adjustment

[0097] 10: splicing and pass Automatically generate design diagrams and provide an explanation of the design intent of the thought process.

[0098] 11: while adjusting the design until the machine-generated design fully meets the requirements;

[0099] 12: return. That is, if you want to generate a series of design drawings, you can repeat the above process and make different changes.

[0100] Figure 3 This is a schematic diagram illustrating multimodal communication between a user and an intelligent agent, according to an exemplary embodiment of this disclosure. Figure 3 As shown, the upper part illustrates the methods of related technologies, and the lower part illustrates the methods provided by embodiments of this disclosure. It may include three steps or stages:

[0101] A. Understanding User Intent (Figure out user intention). Related technologies employ fine-tuning prompts and T2I (text-to-image), where the AI ​​receives user descriptions and directly generates a response result, such as generating a design drawing directly from text. This approach, lacking understanding of the user, often yields unsatisfactory results. This disclosure employs a multimodal large language model capable of autonomous understanding. When the AI ​​receives user descriptions, it doesn't simply generate a design drawing based on the description; instead, it further requests necessary information, such as style, font color, etc. The user then provides this information according to the AI's requests. Through these multiple rounds of interaction, the AI ​​can acquire sufficient design intent information, thereby generating a design drawing more likely to produce a satisfactory result. Here, the generated result is exemplified by the "Cosmic Horse-faced Man" design.

[0102] B. Demonstration to designs. Related technologies require user queries for explanation. For example, if a user inputs "Please explain the design," the AI ​​will reply with "Of course! This is a…," and this is a general explanation that doesn't directly help with subsequent design. However, in this disclosed embodiment, the AI ​​can obtain accurate and sufficient design intent information through multiple rounds of interaction, and can refine the design intent information down to each design element, such as which characters are in the design drawing, what the text of the characters is, what the font is, what the extensions are, whether the background is transparent, and what the object (here referring to the object or object in the design drawing, not the user) is. The AI ​​can draw a separate transparent background image for each design element and has aesthetic awareness. Therefore, on the one hand, it can provide explanatory information without requiring user queries; on the other hand, the provided explanatory information has detailed reasons, thus helping with subsequent design and adjustments.

[0103] C. Iterative Adjustments. Because related technologies edit on a single layer, meaning multiple design elements are on the same layer, when a user instructs to change the text color of a design element, the AI, while editing accordingly, may affect other design elements, adjusting information on other design elements the user does not wish to change. However, this application edits on multiple layers, with each design element corresponding to a separate layer. Therefore, when a user instructs to modify a design element, such as the text color in a design drawing, the target layer containing that design element information can be located first, and then only that target layer can be edited, thus avoiding impact on other design elements. The method provided in this disclosure is compatible with various editing methods and offers higher controllability.

[0104] Based on the structured results obtained in step 1, the multimodal pre-trained model performs semantic feature alignment on the multimodal text and images. For example, using the fine-tuned multimodal pre-trained model, the design intent information obtained in step 1, including descriptive text, optical character images, reference object images, style images, etc., is mapped to the same multimodal feature space.

[0105] Figure 4 This is a schematic diagram illustrating the fine-tuning of a multimodal pre-trained model according to an exemplary embodiment of this disclosure. For example... Figure 4 As shown, the learning process includes a first stage of multimodal representation learning and a second stage of character representation learning. Multimodal representation learning is a technique for learning to embed information from different modalities and their correlations. Character representation learning is the process of converting characters (or sub-characters) into numerical representations that computers can understand and process.

[0106] In the first stage of multimodal representation learning, the input object image is fed into the image encoder 410, and then into the cross attention mechanism in the multimodal encoder 420; the input text is fed into the self attention mechanism in the multimodal encoder 420; and the query vectors are fed into another self attention mechanism in the multimodal encoder 420. The multimodal encoder is an encoder capable of processing and integrating information from different modalities (such as text, images, audio, and video). The self attention mechanism is mainly used for interactions at various positions within a sequence, helping the model capture global information within the sequence. The cross attention mechanism is used for information interaction across sequences or modalities, helping the model establish dependencies between different data sources. Visual features output by the image encoder 410 are extracted through the query vectors, and these features are passed to the input text. In this process, the query vectors act as a bridge, connecting image and text information. The multimodal encoder 420 also includes two feed-forward mechanisms. Feedforward refers to the unidirectional flow of data from the input layer through the hidden layer to the output layer. The multimodal encoder 420 outputs BLIP-2 (Bootstrapping Language-Image Pre-training 2.0) representation learning objectives. That is, the main goal of the first stage is to improve the alignment ability of the model (including the image encoder 410 and the multimodal encoder 420) for image and text representations by pre-training on image-text pairs (including the input object image and the input text).

[0107] Figure 4In this example, the input text is assumed to be: "The image features a cartoon character representing the Earth, depicted in vibrant colors. This character is skating on a skateboard and wearing a cap, with a cheerful expression. The Earth is illustrated with continents and oceans, and the character is also wearing sporty shoes. The background is a light purple color, enhancing the playful and energetic vibe of the illustration. Overall, it conveys a fun and lively atmosphere."

[0108] In the second stage of character representation learning, the image encoder 410 and multimodal encoder 420, which were pre-trained in the first stage, are further pre-trained. Specifically, the input character image (containing text content, which can be arbitrary and not limited in this disclosure) is input to the image encoder 410, and then to the multimodal encoder 420; the character text "T text content T" (character text) and the query vector are also input to the multimodal encoder 420; the output data of the multimodal encoder 420 is then input to the feedforward, which outputs a character prompt embedding, and the character prompt embedding and the text prompt "E golden sunflower E" are input to the text encoder 430. Then, the output of the text encoder 430 and the noisy latent variable are input to the U-Net network 440, which outputs a character image, in which the text content is superimposed with a golden sunflower effect based on the text prompt. Character prompt embedding refers to converting characters or character sequences (i.e., prompts or instructions) into embedded representations. Noisy latent variables refer to those hidden variables in machine learning models that cannot be directly observed and contain noise or uncertainty.

[0109] In this embodiment of the disclosure, the user input includes text characters and images. The model needs to map the images to text. Therefore, a multimodal pre-trained model is required to have the ability to distinguish between text and text characters. Figure 4 The first stage involves mapping pixels to text. The second stage further refines the pixel mapping of text characters. Figure 4 This approach not only maps text and images to a feature space, but also guides the model to learn the relationship between text input and text rendering (character images). Therefore, the first stage connects text and images, and the second stage enhances the connection between text and character images.

[0110] Specifically, cross-attention is used to align features from different modalities (such as images and text), while self-attention focuses on semantic associations within a single modality. Figure 4 In the first stage, the output of the image encoder needs to interact with text features. Therefore, through a cross-attention mechanism, text features can dynamically focus on key regions of the image, achieving fine-grained cross-modal alignment. This embodiment of the present disclosure forces the establishment of cross-modal associations through cross-attention, laying the foundation for subsequent generation tasks (guiding the connection between text and character images).

[0111] In the second stage of the fine-tuning process, U-Net, as a component of the diffusion model, gradually reconstructs the noisy latent variables into the target image through an iterative denoising process. U-Net's multi-layered structure can simultaneously capture local details and global layout.

[0112] In this embodiment, during the fine-tuning of the multimodal pre-trained large model, the ambiguity of text, optical characters, and images in image design is addressed. For example, text can describe image content, generate text, or describe a style. Similarly, an image can be a stylized version of the image itself or a reference style for optical character generation. Therefore, multiple special markers are designed to distinguish different situations and alleviate the ambiguity of language expression. Then, different modalities are mapped to the same feature space for convenient interaction and unified generation. Different special markers distinguish input types, guiding the model to learn task-specific features. For example, does "text" represent text characters or text content? For instance, if a user inputs "apple," does it provide an image of the characters for "apple" or generate an image of an apple?

[0113] For example, it can include Figure 4 The example provided uses "T Text Content T" to indicate that the text content is character text, and "E Golden Sunflower E" to indicate that it is a text prompt. For example,<b*> and<\b*> are used to indicate bold text.<i*> and<\i*> is used to indicate italic text.<u*> and<\u*> are used to represent underscores, etc.

[0114] In this embodiment of the disclosure, a transparent base map of each design element is generated based on the multimodal feature sequence in the multimodal pre-trained model. For example, an end-to-end transparent map generation model can be retrained based on a generation framework, such as a diffusion-based or flow-based generation model, and a large number of transparent base map data samples, thereby obtaining a hierarchical representation of each individual design element.

[0115] Figure 5 The process of collecting a training dataset for training a model that generates transparent graphs is illustrated. For example... Figure 5As shown, (a) illustrates the decomposition process of graphic design, and (b) demonstrates the generation of a structured JSON file from the decomposed graphic design. (c) shows examples of collected graphic design images. These images can be processed as shown in (a) to obtain the structured JSON file shown in (b). Then, the structured JSON file is converted into a multimodal feature sequence using a pre-trained visual language model and input into a transparent graph model to generate a transparent background map of the corresponding design element information. This background map is then compared with the transparent background map of the decomposed design element information (as labels) in (a), and the loss function is calculated to train and obtain the transparent graph model.

[0116] Specifically, taking a graphic design image containing various design elements such as character 1 "What do you want to eat?", character 2 "bread", object 1 - an image of bread, and background as an example, the image is segmented into transparent background images, each containing a single design element, using a segmentation model. These are used as transparent background image data samples, meaning the training dataset for the transparent image generation model is layered. To organize the graphic design down to the element level, a structured JSON file containing the graphic design image is obtained. This can be achieved by using a detection sub-model to detect the bread image, extract the bread image layout, and using a caption sub-model (which is an image-to-text model) to recognize and generate captions for the graphic design image, such as "This is for product advertising. It shows a natural style. There are two pieces of bread on the center." This obtains information about each design element and then generates a structured JSON file based on this information.

[0117] For example, suppose the resulting structured JSON file looks like this:

[0118] "overall": / / the whole

[0119] [{

[0120] "Target": "Product Design", / / Target: Product Design

[0121] "Style": "Natural", / / Style: Natural

[0122] "layout": "imgpath" / / Layout, imgpath actually points to the path / address of an image file.

[0123] }],

[0124] “items”: / / design elements

[0125] [{"background":[{ / / background]}

[0126] "src": "imgpath", / / source

[0127] "type": "background", / / type

[0128] "description": "textured white", / / Description: Textured white

[0129] "width": "1024", / / width

[0130] "height": "1024" / / Height

[0131] ]}

[0132] “item2”:[{

[0133] “src”:“imgpath”,

[0134] "type":"img",

[0135] "description": "two pieces of bread", / / Description: Two pieces of bread

[0136] "angle": "0", / / Angle

[0137] "width": "626",

[0138] "height": "556"

[0139] ]}

[0140] “item3”:[{

[0141] “src”:“imgpath”,

[0142] …

[0143] Figure 6 This is a schematic diagram of a design drawing generated according to an exemplary embodiment of this disclosure. The design drawing generated using the method provided in the embodiments of this disclosure is, for example, as shown below. Figure 6As shown in (a), it includes four design elements: an image of a glass bottle, the text "A glass bottle," the bottle itself, and a smiley face on the bottle. These four design elements are layered, meaning each design element corresponds to a separate layer. For example, as... Figure 6 As shown in (b), the image of a glass bottle corresponds to layer 601, the text "a glass bottle" corresponds to layer 602, the body of the glass bottle corresponds to layer 603, and the smiley face on the glass bottle corresponds to layer 604. This allows users to easily edit the design elements corresponding to each individual layer according to their design needs.

[0144] In this embodiment of the disclosure, based on the individual transparent background images of each design element obtained in step 3 and the reference layout diagram in step 1, an aesthetically pleasing splicing model is used to splice all the design elements to obtain the final design drawing. For example, as shown in the figure... Figure 6 As shown in (c-1), (c-2), and (c-3), it can be used Figure 6 The image shown in (a) is used to generate different design drawings after coordination.

[0145] This disclosure provides an autonomous intelligent agent for graphic design, moving towards general artificial intelligence. This disclosure relates to automated graphic design technology, and more specifically, to intent recognition and image generation in a series of graphic design processes.

[0146] Artificial General Intelligence (AGI) marks a significant leap forward in the field of artificial intelligence. Its goal is to build AI systems capable of performing a wide range of intelligent tasks, simulating human cognitive abilities and demonstrating superior performance across diverse domains and tasks. Advances in AGI have profound implications for driving technological innovation, promoting economic development, leading social change, and stimulating cultural innovation. In the field of artistic creation, AGI's influence is particularly prominent. It not only replicates an artist's style to create works but also pioneers entirely new art forms, expanding the boundaries of art. By absorbing and analyzing a vast amount of artwork, AGI gains insights into diverse art movements and techniques, thereby creating unique and innovative artistic masterpieces. This creative model not only provides artists with a rich source of inspiration but also helps them explore unknown creative paths and technological means.

[0147] In related technologies, the graphic design process is complex and iterative, typically encompassing the following key stages: First, the research and conceptualization stage, where designers conduct market research to clarify the design direction and initial concepts, although these concepts are often broad and vague in the early stages. Next, the creative brainstorming stage, where designers develop ideas based on previous research findings. Following this, the design development stage, where designers use specialized software to refine the concepts into concrete design sketches and begin creating and perfecting design elements. Then comes the feedback and revision stage, where the initial design draft undergoes continuous adjustments and optimizations. Finally, in the final design stage, after several rounds of revisions, the designer presents the final design, including the final layout of all graphic elements and necessary detail finishing touches.

[0148] With the rapid advancement of artificial intelligence technology, especially the rise of large language models, we have witnessed a shift from automating simple tasks to handling complex creative tasks. Nevertheless, most graphic design tools in related technologies remain limited to assisting design, failing to fully realize the ability to translate user intent into concrete designs, particularly in interpreting vague or abstract design instructions. This disclosure, through a multimodal large language model and an instruction-based proactive guidance mechanism, obtains detailed user design intent information through multiple rounds of interaction with the user, thereby preventing the user from missing certain information. Furthermore, it can elucidate the abstract concepts initially proposed by the user through continuous communication. This communication is proactively guided by instructions, allowing the multimodal large language model to know which direction to ask questions, actively communicating with the user, and refining abstract concepts into concrete design elements.

[0149] Furthermore, tools in related technologies often lack design transparency and fail to elucidate the internal logic of design decisions, which is crucial for verifying and trusting AI-generated designs. The embodiments disclosed herein, however, provide detailed and reasonable explanatory information.

[0150] Figure 7 This is a schematic diagram of the structure of a graphic design apparatus according to an exemplary embodiment of this disclosure. Figure 7 As shown, the graphic design apparatus 700 provided in this embodiment includes an interaction unit 710, a mapping unit 720, a processing unit 730, and a generation unit 740.

[0151] The interaction unit 710 interacts with the object using a multimodal large language model to obtain the object's design intent information and converts the design intent information into design element information. The mapping unit 720 maps the design element information to the same multimodal feature space using a pre-trained visual language model to obtain a multimodal feature sequence of the design element information. The processing unit 730 processes the multimodal feature sequence using a transparency image generation model to generate a transparent background image of the design elements. The generation unit 740 stitches and integrates the transparent background images of the design elements to generate a design drawing. Figure 7 Other aspects of the embodiments can be found in the other embodiments described above.

[0152] Figure 8 A schematic structural diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. Figure 8 The illustrated electronic device 800 includes a processor 810, which can call and run computer programs from memory to implement the methods in any embodiment of this disclosure. Optionally, as... Figure 8 As shown, the electronic device 800 may further include a memory 820. The processor 810 can call and run computer programs from the memory 820 to implement the methods in the embodiments of this disclosure. The memory 820 may be a separate device independent of the processor 810, or it may be integrated into the processor 810. Optionally, as... Figure 8 As shown, the electronic device 800 may further include a transceiver 830, which the processor 810 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices. Optionally, the processor 810, the memory 820, and the transceiver 830 can achieve bidirectional communication with each other through a communication bus 840.

[0153] It should be understood that the processor in this embodiment of the disclosure may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by software instructions.

[0154] This disclosure also provides a computer-readable storage medium for storing a computer program. The computer program causes a computer to execute corresponding flows in the various methods of the embodiments of this disclosure; for simplicity, these details are not elaborated here. This disclosure also provides a computer program product, including computer program instructions. These computer program instructions cause a computer to execute corresponding flows in the various methods of the embodiments of this disclosure; for simplicity, these details are not elaborated here. This disclosure also provides a computer program. When this computer program is run on a computer, it causes the computer to execute corresponding flows in the various methods of the embodiments of this disclosure; for simplicity, these details are not elaborated here.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least zero, such as two, three, etc., unless otherwise explicitly specified. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0156] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

Claims

1. A graphic design method, characterized in that, include: By interacting with the object through a multimodal large language model, the design intent information of the object is obtained, and the design intent information is converted into design element information; The design element information is mapped to the same multimodal feature space by a pre-trained visual language model to obtain the multimodal feature sequence of the design element information; The multimodal feature sequence is processed by a transparency graph generation model to generate a transparent base map of the design elements; The transparent background images of the design elements are stitched together to generate the design drawing.

2. The method as described in claim 1, characterized in that, By interacting with objects through a multimodal large language model, the design intent information of the objects is obtained, and the design intent information is converted into design element information, including: The first design intent information sent by the object is received through the multimodal large language model, and the first design intent information is converted into first design element information. Fill the corresponding position of the graphic design element template with the information of the first design element; If an unfilled design element is detected in the graphic design element template, the multimodal large language model is triggered to send a query message corresponding to the unfilled design element to the object. The second design intent information returned by the object in response to the query information is received through the multimodal large language model, and the second design intent information is transformed into second design element information. The design intent information includes the first design intent information and the second design intent information, and the design element information includes the first design element information and the second design element information.

3. The method as described in claim 2, characterized in that, The second design intent information is converted into second design element information, including: If the second design intent information indicates that the unfilled design element is empty, then the second design element information corresponding to the unfilled design element is generated based on the first design element information; The second design element information is sent to the object through the multimodal large language model; In response to the object's feedback information regarding the second design element information, the second design element information is determined.

4. The method as described in claim 1, characterized in that, By interacting with objects through a multimodal large language model, the design intent information of the objects is obtained, including: By interacting with the object through the multimodal large language model, the preliminary design intent information of the object is obtained, and the preliminary design intent information is converted into preliminary design element information; Based on the preliminary design element information, a matching target subgraph is retrieved from the knowledge graph of the graphic design domain. Preliminary adjustment information for the preliminary design intent information is generated based on the target sub-graph; The design intent information is obtained by responding to the object's feedback on the preliminary adjustment information.

5. The method as described in claim 4, characterized in that, Also includes: Based on the knowledge graph of the graphic design domain, the preliminary adjustment information, and the feedback information, explanatory information for the design drawing is generated.

6. The method as described in claim 1, characterized in that, The transparent background images of the design elements are stitched together to generate a design drawing, including: Based on the knowledge graph of the graphic design domain, the transparent background image of the design element is adjusted; The design drawing is generated based on the transparent base map of the adjusted design elements.

7. A graphic design device, characterized in that, include: An interaction unit is used to interact with an object through a multimodal large language model, obtain the design intent information of the object, and convert the design intent information into design element information. The mapping unit is used to map the design element information to the same multimodal feature space through a pre-trained visual language model to obtain the multimodal feature sequence of the design element information; The processing unit is used to process the multimodal feature sequence through a transparency map generation model to generate a transparent base map of the design elements; The generation unit is used to stitch together and integrate the transparent background images of the design elements to generate a design drawing.

8. An electronic device, characterized in that, include: One or more processors; A memory configured to store one or more programs that, when executed by one or more processors, cause the electronic device to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on a computer, it causes the computer to perform the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images