System and method for automatically generating design

Abstract

A computer-implemented method for automatically generating a design is described herein. The method includes receiving an input prompt for generating the design and generating a design outline based on the input prompt. The design outline includes a style descriptor for the design. The method also includes retrieving a design template based on the design outline. The design template includes placeholder design elements. The method also includes retrieving at least one design element of the placeholder element. The design is generated by populating the placeholder element with the design element.

Description

[0001] Incorporation by Reference

[0002] The entire contents of Australian provisional application No. 2023210536 filed on July 31, 2023 for “Systems and methods for processing designs” are incorporated herein by reference. Technical Field

[0003] Aspects of the present disclosure relate to systems and methods for automatically generating designs. Background Art

[0004] There are computer applications for creating and processing designs. Some such applications can provide users with the ability to create designs in different formats. Generally, such applications allow users to create designs by, for example, creating a page and adding design elements to the page. Such applications can provide multiple design templates in various design categories to help users create designs.

[0005] While computer tools exist for manually generating such designs, generation of the designs is often a time-consuming and sometimes complex task requiring manual generation of textual content and retrieval of appropriate media content tailored to the design theme or topic.

[0006] Therefore, smarter computer applications are needed to help users create designs. Summary of the invention

[0007] A computer-implemented method for automatically generating a design is described herein, the method comprising: receiving an input prompt for generating the design; generating a design outline based on the input prompt, the design outline comprising a style descriptor for the design; retrieving a design template based on the design outline, the design template comprising a placeholder design element; retrieving at least one design element of the placeholder element; and generating the design by filling the placeholder element with the design element.

[0008] The present invention also describes a computer processing system comprising one or more computer processing units; and a non-transitory computer-readable medium storing instructions, which, when executed by the one or more computer processing units, cause the one or more computer processing units to perform the computer-implemented method described above.

[0009] Also described herein is a non-transitory storage device storing instructions executable by one or more computer processing units to cause the one or more computer processing units to perform the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the attached picture:

[0011] Figure 1 is a diagram depicting an example network environment for performing various features of the present disclosure.

[0012] Figure 2 is a block diagram of an example computer processing system.

[0013] Figure 3 is a process flow diagram depicting an example method for automatically generating one or more candidate designs in response to an input prompt.

[0014] Figure 4 is a process flow diagram depicting an example method for automatically generating a design outline for one or more candidate designs.

[0015] Figure 5 is a process flow diagram depicting an example method for retrieving or selecting a design component for one or more candidate designs based on a design outline.

[0016] Figure 6 is a process flow diagram depicting an example method for generating textual content for one or more candidate designs.

[0017] Figure 7 is a process flow diagram depicting an example method for generating and outputting one or more candidate designs.

[0018] Figure 8 is an example user interface displaying candidate designs according to some aspects of the present disclosure.

[0019] Fig. 9 is a sample user interface for receiving user input.

[0020] Although various modifications and alternative forms are possible in this specification, specific embodiments are shown and described in detail by way of example in the drawings. However, it should be understood that the drawings and detailed description are not intended to limit the invention to the specific forms disclosed. The present invention is intended to cover all modifications, equivalents and alternatives that fall within the scope of the invention as defined by the appended claims. DETAILED DESCRIPTION

[0021] In the following description, many specific details are set forth to provide a thorough understanding of the present invention. However, it is apparent that the present invention can be implemented without these specific details. In some cases, well-known structures and devices are shown in block diagram form to avoid unnecessary confusion.

[0022] As discussed above, there are computer applications for creating and managing designs. Such applications can provide users with mechanisms to create designs, edit designs by adding content to designs, and output designs in various ways (e.g., by saving, displaying, printing, publishing, sharing, or otherwise outputting designs). As also discussed above, applications can provide users with the ability to create and manage designs in different formats, such as cards, posts, posters, presentations, documents, etc.

[0023] As discussed further above, there are many scenarios in which a user may wish to create a design, such as to display on a theme. At a general level, creating a design involves a user selecting a design template. Generally speaking, a design template includes a plurality of customizable / placeholder elements, such as text boxes, image boxes, shapes, banners, borders, color themes, etc., which are configured to receive design elements from a user. Therefore, a design template can provide a structure to locate the content of a design for display. Some computer applications may include different types of design templates suitable for different types of designs, such as design templates for presentations, design templates for posters, design templates for invitations, design templates for social media posts, etc. Depending on the type of design a user wishes to create, a user can select a design template from the available templates suitable for the design type, and then customize the design template by adding design elements to the design template. Design elements include, for example, text content (e.g., a sequence of one or more characters) and shape content. Shape content can include media content (e.g., images, videos, audio clips) and fill content (e.g., shapes of solid colors or color gradients). The user then manually deletes one or more placeholder elements, adds one or more placeholder elements, moves one or more placeholder elements around, and adds design elements to the placeholder elements until the user is satisfied with the final design. It should be understood that in the case where the application provides a large number of design templates, such as over a million design templates, the user's browsing and selection of design templates may be extensive. Once the user selects a template, traditionally, the user still needs to manually generate text content and retrieve media time to populate such selected template. This can be tedious and time-consuming work for the user, and can often result in poorly created designs, such as designs that do not contain suitable media content or filler content that matches the text content of the design, or the design template is not particularly suitable for the design.

[0024] Aspects of the present disclosure solve one or more problems in the traditional design generation process by providing a system and method for automatically and intelligently generating one or more candidate designs based on an input prompt. Specifically, the system and method are configured to analyze the input prompt and 1) select one or more design templates, 2) select or generate design elements for each of the selected one or more design templates, and 3) generate one or more candidate designs by transferring the selected or generated design elements to the corresponding design template.

[0025] To select one or more design templates, the currently disclosed systems and methods determine one or more design categories (e.g., posts, social media posts, presentations, invitations, etc.) and design styles (e.g., vibrant, vintage, elegant, etc.) based on the input prompt. Then, one or more design templates are selected for the design using the identified one or more design categories and design styles.

[0026] Once the system and method selects a design template, the system and method identify placeholder design elements for the design template and automatically generate or select content for the placeholder design elements based on the input prompt and the identified design style. For example, if the design template contains two text-based placeholder design elements for a title and a subtopic and one placeholder design element for an image, the system and method automatically generates the text for the text-based design elements based on the input prompt and / or the design type and design style.

[0027] Additionally, the systems and methods may automatically generate one or more media queries for the image placeholder element based on the input hint, and then search the media library for media items that match the media queries.

[0028] Finally, the system and method transfer the generated design elements into the placeholder elements of the selected design template to generate one or more candidate designs. These candidate designs can then be displayed on the user device of the user who generated the initial input prompt.

[0029] In some embodiments of the present disclosure, a generative machine learning (ML) model may be utilized to determine a design style (also referred to herein as a style descriptor) and generate a media query based on the input prompt. In the present disclosure, the design style and media query are collectively referred to as a design outline. In addition, a machine learning model may be utilized to generate text content for a text-based placeholder element in a selected design template.

[0030] In some embodiments, a first input prompt is generated and provided to a generative ML model, which generates a design outline based on the first input prompt. One or more design templates are selected based on the design outline. A second set of input prompts (e.g., based on the selected design templates) is then generated and provided to the ML model, which generates text content for a text-based placeholder of the selected design template based on each second prompt in the second set of prompts.

[0031] A generative ML model can be trained using the configuration data and some lens examples to accurately generate design outlines and text content.

[0032] The media query generated by the ML model can be used to search for media items in the media library and select the media items for the corresponding page. The style descriptor generated by the ML model can be used to search for design templates in the template library and select the template for generating the design. The style descriptor can also be used to search for style attributes in the style attribute library and select the style attributes applied to the design elements in the design template.

[0033] Example Design Data Structure

[0034] Data about designs that have been (or are being) created can be stored in various formats. The following sections describe the data structures used to generate and store designs in the present disclosure. The described data structures and fields are provided by way of example. Depending on the embodiment, alternative design formats utilizing the same, additional, fewer and / or alternative fields can be used, and the processing described herein can be applied to alternative formats. In addition, the fields described about a given data structure can be stored in one or more alternative data structures (e.g., data structures across multiple links).

[0035] The systems and methods described herein utilize design plan descriptors when generating candidate designs. Design plan descriptors are temporary records that store data retrieved, generated, and selected during the design process, which are ultimately used to create candidate designs. To aid understanding, Table A below shows a partial example of a design plan descriptor.

[0036]

[0037]

[0038] Table A: Example Design Plan Descriptor

[0039] In this example, the fields of the design plan descriptor include: a plan identifier (which uniquely identifies a design plan); an input prompt (e.g., a prompt entered by a user to begin the design creation process); a design category; a style descriptor; a media query; a design template (e.g., an identifier of one or more design templates selected for a candidate design), a color palette (e.g., an identifier of one or more color palettes selected for a candidate design), a font palette (e.g., an identifier of one or more fonts selected for a candidate design), text content (e.g., text content generated for a candidate design), and template analysis data (e.g., indicating the type of text placeholder elements present in a design template). The design plan descriptor will be described and utilized throughout this disclosure.

[0040] In this context, data about a particular design is stored in a design record. A design record defines certain design-level properties and contains design metadata, design format data, and design elements. In this example, the format of each design record is a device-independent format, including a set of key-value pairs. To aid understanding, Table B below shows a partial example of a design record format.

[0041]

[0042]

[0043] Table B: Example Design Record

[0044] In this example, the design-level attributes include: a design identifier (which uniquely identifies a design); design dimensions (e.g., a default width and height of a design); design type (e.g., an indicator of the design type, which can be used for searching and / or sorting purposes); design name (e.g., a string defining a default name or a user-specified name for a design); design owner (e.g., an identifier of a user or group that owns or creates a design); last edit time (e.g., a timestamp indicating when the design was last edited); background (data indicating any page background that has been set, such as an asset identifier for an image that has been set as a page background, a value indicating a specific color for a solid background fill, or data indicating an alternate background); and element data (discussed below). Additional and / or alternative design-level attributes may be provided, such as attributes regarding creation date, design version, design permissions, and / or other design-level attributes.

[0045] The element data is a set (an array in this example) of element records. Each element record defines an element (or set of grouped elements) that has been added to the design. In this example, the position of the element record in the design element array is used to identify the element and also determine the depth or z-index of the element (or group of elements) in the design (e.g., the element at array index n is above the element at array index n-1 and below the element at array index n+1). However, element depth may be handled alternatively, for example, by storing the depth as an explicit element attribute.

[0046] In general, an element defines content that has been (or will be) added to a design, such as by copying / importing content from one or more asset libraries (e.g., a media library, a style library, etc.); by drawing / creating using one or more design tools (e.g., a text tool, a line tool, a rectangle tool, an ellipse tool, a curve tool, a freehand tool, and / or other design tools); or by otherwise adding content to a design. Various element types may include position data that defines the position of the element on the page to which it belongs. In this example, the position of the element is defined by an (x, y) coordinate pair. Various element types may also include size data that defines the size of the element. In this example, the size of the element is defined by a width value and a height value. Various element types also include size data that defines the size of the element. In this example, the size of the element is defined by a width value and a height value. Various element types also include rotation data that defines the rotation of the element. In this example, the rotation of the element is defined by a rotation value that defines the degree by which the element is rotated (e.g., 0 = no rotation). In this embodiment, any rotation applied to the element is around the center of the element.

[0047] Depending on the system in question, different types of design elements may be provided. For example, the following types of design elements may be provided: text elements; shape elements (e.g., media items and other fill-type content) and / or other types of elements. Each element will typically contain a content item, metadata, and format and attribute data. For a given design element, the content defined by the corresponding content item has the same type as the field type corresponding to the content item. For example, a content item corresponding to a text type design element may define text content, while a content item corresponding to an image type design element defines image content. As will be appreciated, different attributes may be associated with different element types. For example, any element that stores visual content (e.g., images, videos, text, etc.) will typically be associated with position and size data, while such data may be unrelated to an element that stores audio media. Therefore, different element record formats (with different attributes) may be used for different element types.

[0048] In the present disclosure, shape elements are used to store visual content, such as images. In this way, image type shape elements are used to store image type content items. For example, image type content items can be raster images (e.g., digital photos, renderings or alternative raster images) or vector graphics images. For example, an element record of an image type shape element is shown in Table C.

[0049]

[0050] Table C: Example Design Element Record

[0051] Although the above element records are about image type elements (e.g., indicating that the fill is a raster graphic), shape type elements can also accommodate different types of media items and fills, such as graphic fill types (indicating that the fill is a vector graphic), color fill types (indicating that the fill is a solid color or a color gradient), video fill types (indicating that the fill is a video), and / or other fill types.

[0052] In addition to the fields described above, a text element (which is used to display text content) may include a text content field, which contains the actual text (e.g., a set of one or more characters) and an attribute field (which defines how the text characters are displayed). Attributes may include data such as font type, font color, font style, font size, and / or other formatting data.

[0053] In certain embodiments, text elements may also include text level data (or be associated with) for one or more hierarchical levels defining text characters. Text level may be represented in various ways. In certain embodiments, a numerical level is used, wherein a lower value indicates a text of relatively greater importance. For example, a level 0 text may correspond to a pre-heading, a level 1 text may correspond to a title, a level 2 text may correspond to a sub-title, and a level 3 text may correspond to the body, etc. Alternatively, a text level may be specified as a type, such as a pre-heading type, a title type, a sub-title type, a body type, etc. The text level may be determined in various ways. In certain embodiments, a text level may be clearly defined for any text. For example, a design template including a placeholder text element may clearly include a default text level for each text element. In an alternative embodiment, a text level may be automatically determined by processing a given design (or design template). Generally speaking, this may involve extracting text content from a design and analyzing text to determine different text levels based on, for example, certain key terms in the text content, certain format attributes, and / or relative text (x, y coordinates) positions on a page.

[0054] Design templates

[0055] Design templates of various different design category types can be stored in the template library. Each design template is stored as a record in the library. Design categories may include flyers, posters, social media posts, social media stories, logos, presentations and videos, video thumbnails, invitations. Additional and / or alternative categories are also possible. Each design template can be associated with various preset design attributes (e.g., colors, fonts, etc.).

[0056] In this context, data about a particular design template is stored in a design template record. A design template record defines certain design-level properties and contains template metadata, template format data, and placeholder design elements. In this example, the format of each design template record is a device-independent format, including a set of key-value pairs. To aid understanding, Table D below shows a partial example of a design template record format.

[0057]

[0058]

[0059] Table D: Example Design Record

[0060] In this example, the design-level attributes include: a template identifier (which uniquely identifies a design template); a template style (which indicates a particular design style of the template), a design size (e.g., a default width and height of a design); a template type (e.g., an indicator of a type or category (or subcategory) of a design, which can be used for search and / or sorting purposes); a template name (e.g., a string defining a default name or user-specified name of a design); a template owner (e.g., an identifier of a user or group that owns or creates a design); style attributes (data indicating any style attributes of a design, such as a color palette, a font palette, and / or other style attributes); and placeholder element data (discussed above). Additional and / or alternative design-level attributes may be provided, such as attributes regarding a creation date, a design version, design permissions, and / or other design-level attributes. The various design-level attributes may be used as tags or terms (e.g., as metadata) to facilitate classification and searching. For example, templates may be classified and / or searched based on template style, template type, template name, size, and combinations thereof. Templates may also or alternatively be associated with other attributes (eg, a keyword attribute) that may be used to store one or more keywords, which may also or alternatively be used to organize and / or search / browse templates.

[0061] As shown in Table C, the design template contains various types of placeholder elements configured to receive corresponding types of design elements. The placeholder element can be positioned at (or configured to position the source element at) a predetermined position on the page, for example, at a specific x and y coordinate and / or occupy a specific pixel range of the page with a predetermined resolution. The placeholder element can also contain style attributes, such as color, font, etc., which can be applied to the design element when filled into the placeholder element. A wide variety of design templates, placeholder elements, style attributes and arrangements thereof are possible. For example, in some embodiments, instead of (or in addition to) design-level style attributes, style attributes can be stored as placeholder element data for the template.

[0062] Templates can include text placeholder elements (e.g., titles, subtitles, body text, etc.) and shape placeholder content elements (e.g., media items, shapes, etc.). Text placeholder elements can include placeholder text and can be configured to receive corresponding text of a specific length (number of characters, words, or sentences) and / or a specific form (paragraphs, bullet points, etc.). Each such text placeholder element can also include one or more default style attributes for adjusting font formatting, including font type, font color, and / or font size. Image placeholder elements can be configured to receive images of a specific size, resolution, and / or aspect ratio. Shape placeholder elements can be configured to receive shapes or fill elements, such as frames or banners. Shape placeholder elements can also include style attributes, such as fills or gradients of specific colors. Any such style attributes can be default style attributes that can be overwritten by applying alternative style attributes.

[0063] As an example, a design template could contain the following placeholder elements:

[0064] A leader placeholder element, which contains the placeholder text "Put your leader here", right-aligned, Calibri (light), size 12;

[0065] A title placeholder element containing the placeholder text "Please put your main point or topic here", bold, centered, Calibri, size 18;

[0066] A subheading placeholder element containing the placeholder text "Please briefly describe what you intend to discuss", underlined, left-aligned, in Calibri, size 16 font;

[0067] A body placeholder element containing the placeholder text "Discuss your topic here", left-aligned, Calibri, size 14 font;

[0068] An image placeholder element containing a shape element for receiving an image of a specific resolution range that accommodates the placement of a text placeholder element;

[0069] ●And a Fill placeholder element that contains a shape element used to define a banner with a specific color or color gradient fill.

[0070] While the above examples include single instances of a lead-in, title, subtitle, and body text, alternative templates with alternative text placeholder elements may also be used. For example, a template may include multiple instances of placeholder body text, or alternatively may include no placeholder text elements and only one or more shape placeholder elements. A design template may provide placeholder shape elements for receiving a fill or image, or may be pre-populated with images and / or shapes, for example, from a corresponding asset library.

[0071] Design assets

[0072] As used herein, design assets refer to pre-existing design components stored, for example, in one or more asset libraries. Examples of design assets include media items and design styles (e.g., defined color sets or color palettes, font types, and / or other assets / asset parameters). All design assets can be stored as records in a single library or as records in corresponding sub-libraries, such as a media library, a font library, a color library, etc.

[0073] Color style attributes may be stored in a variety of formats. As an example, one or more color style attributes may be stored as a color palette. Generally, a predefined color palette defines one or more (usually complementary) sets of colors. For example, a given predefined color palette may be an array of RGB (or other color space) color values ​​corresponding to a predetermined number of colors (usually four to six colors).

[0074] Color palettes and individual colors may include (or be associated with) tags or terms (e.g., as metadata) that facilitate categorization and searching. Examples of tags for colors and color palettes may include "bold," "vibrant," "vintage," "organic," "loud," "soft," "professional," "modern," and the like. A large number of colors, color palettes, tags, and combinations thereof are possible.

[0075] The color palette may also include (or be associated with) data for associating the colors in the palette with specific shapes or shape elements and / or specific text elements having specific text hierarchy levels. As an example, the association with the colors in the palette may be explicitly configured in the metadata, e.g., the color palette may include "color 1" applicable to text elements at text hierarchy level 1. Additionally or alternatively, in some embodiments, color tags in the color palette may be used to associate the corresponding colors with text elements. For example, a "bold" tagged color in the color palette may be associated with text elements having a bold font and / or text hierarchy level 1 text elements (e.g., headings) in the design template that are intended to be bold.

[0076] Many alternative color style properties and color palette structures are possible.

[0077] Font style properties applicable to design elements may be stored in a variety of formats. As an example, one or more font style properties may be stored as a font palette. A predefined font palette defines one or more (usually complementary) font sets. For example, a given predefined font palette may be an array of (usually two to four) font types (e.g., Times New Roman, Helvetica, Calibri, etc.) and / or font sets (e.g., Arial, Arial Black, Arial Narrow).

[0078] Font palettes and individual fonts may include (or be associated with) tags or terms (e.g., as metadata) that facilitate classification and searching. Examples of tags for fonts and font palettes may include "bold," "emphasized," "calligraphic," "handwritten," "vintage," "organic," "loud," "soft," "professional," "modern," and the like. Font palettes may also include fonts and / or corresponding font metadata and other font attributes (e.g., font size, bold, underline, italics) for each font. A large number of fonts, font palettes, attributes, tags, and combinations thereof are possible.

[0079] The font palette may also include (or be associated with) data for associating fonts in the palette with specific text elements having specific text hierarchy levels. As an example, associations with fonts in the font palette may be explicitly configured in metadata, e.g., a font palette may include "font 1" applicable to text elements at text hierarchy level 1. Additionally or alternatively, in some embodiments, font tags in the font palette may be used to associate corresponding fonts with text elements. For example, an "emphasis" tagged font in a font palette may be associated with text elements having a bold font and / or text hierarchy level 1 text elements (e.g., headings) in a design template that are intended to be emphasized.

[0080] Many alternative font style properties and font palette structures are possible.

[0081] In addition to design-related data, data related to design usage can also be maintained and stored in various formats. In one example, the systems and methods disclosed herein can store usage data about search terms for a search and the categories of corresponding designs or design templates selected, viewed, or published in connection with the search.

[0082] To help understand the following description, a partial example of a usage metrics record for a design template category for a search query may be as follows:

[0083]

[0084] Table E: Example usage metrics

[0085] In this example, a usage metric maps a search query (and / or terms of a search query) to the proportion of design template categories of published designs that were viewed, selected, or published from such a search query. Each record in the table contains specific search terms used to search for design templates, as well as the template categories that were selected after searching using these search terms and ultimately published as designs. For each design category, in each record, the proportion of publications in that category is also contained. In this way, the usage metric provides a category distribution as an indication of the probability of using a design template of a particular category when the template search contains specific terms. Although this example illustrates three design categories of published designs, more or less is possible. In addition, many additional and / or alternative usage metrics and data structures are also possible. For example, although presented above in the form of a simplified table, the usage metrics can be stored as a JSON file.

[0086] Usage metrics data may be stored and maintained (e.g., regularly updated) so that usage of design assets (e.g., design templates, media items, and style attributes) may be analyzed. A number of implicit conclusions may be drawn from analysis of usage metrics. For example, if templates belonging to a particular design category are selected more frequently than templates of another category with respect to a particular search term, it may be concluded that the particular design category of the template is more relevant to the search term.

[0087] Thus, usage metrics are maintained so that the use of images / graphics and / or design templates can be analyzed. For example, usage metric data can also be used in the present disclosure to determine the expected category of a design template based on an input prompt. These functions will be described in more detail in the following sections.

[0088] Example System

[0089] The technology disclosed herein is described in the context of a digital design platform that is configured to facilitate various operations related to digital design. In the context of the present disclosure, these operations relevantly include automatically creating designs. The digital design platform can take various forms. In the embodiments described herein, the digital design platform is a client-server type platform (e.g., one or more client applications and one or more server applications that interoperate to perform the described technology). However, the technology described herein can be performed (or adapted to be performed) by an independent digital design platform (e.g., an application or set of applications that runs on a user's computer processing system and performs the technology described herein without requiring server-side operations).

[0090] Figure 1 An example networked environment 100 is depicted in which various features of the present disclosure may be implemented. Networked environment 100 includes server system 110, ML system 130, and client system 140, which operate together to perform the operations described herein.

[0091] Systems 110-140 communicate with each other via one or more communication networks 150 (eg, the Internet). For example, client system 140 communicates with server system 110 via the public Internet, and server system 110 may communicate with ML system 130 via a local area network or a public area network.

[0092] The server system 110 is a system entity that hosts one or more computer applications and / or content. The server system 110 may include one or more server computing systems or nodes for hosting server applications 112 and one or more storage devices (e.g., data storage device 119) for storing application-specific data. Examples of server applications hosted by the server system 110 include digital design applications (e.g., Canva Design).

[0093] The server system 110 can be executed to provide a client application endpoint accessible via the communication network 150. In some examples, the server system 110 is a network server that provides services to web browser clients and receives and responds to HTTP requests. In another example, the server system 110 is an application server that provides services to local client applications and is configured to receive, process, and respond to specifically defined API calls received from those client applications. The server system 110 may include one or more network server applications and / or one or more application server applications, allowing it to interact with web and local client applications.

[0094] Although a single server architecture is described herein, it should be understood that the server system 110 can be implemented using alternative architectures. For example, in some cases, a cluster architecture can be used, in which multiple server computing instances (or nodes) are instantiated to meet system requirements. Communication between the applications of the server system 110 and the computer processing system can be carried out by any appropriate means, such as direct communication or networked communication through one or more local area networks, wide area networks, and / or public networks (if necessary, using security logic overlays, such as VPNs). In contrast, in the case of a small business with relatively simple requirements, the server system 110 can be a stand-alone implementation (i.e., a single computer that is directly accessed / used by a client).

[0095] The server application 112 (and / or other applications of the server system 110) in conjunction with the client application 142 facilitates various functions related to digital design. For example, these functions may include, for example, design creation, editing, organization, searching, storage, retrieval, viewing, sharing, publishing, and / or other functions related to digital design. The server application 112 may also facilitate additional related functions, such as user account creation and management, user group creation and management, and user group permission management, user authentication, and / or other server-side functions.

[0096] To perform the functions described herein, the server application 112 includes a plurality of software modules that provide various functions and interoperate to automatically generate designs. These modules will be discussed below and include: a category determination module 113; a machine learning module 114; a hint generation module 115; and a design generation module 116.

[0097] In this embodiment, the category determination module 113 is configured to determine one or more design categories based on the input prompt. For example, the category determination module 113 can process the input prompt and determine the design category associated with (or used for association with) the input prompt. The category determination module 113 can process the input as a complete string, individual words (e.g., a set of characters delimited by spaces), and / or groups of words or sub-queries within the input.

[0098] The ML module 114 is configured to communicate with the ML system 130 via the network 150. Specifically, it is configured to provide one or more input prompts to the ML system 130 and receive one or more outputs from the ML system 130. The outputs may include a design outline and design content.

[0099] The prompt generation module 115 is configured to receive input prompts and generate design outline prompts and design content prompts for the ML system 130 .

[0100] The design generation module 116 is configured to receive design outlines, design categories, and design content from various modules, and select design assets (e.g., design templates, fonts, and colors) based on the design outlines and categories, and also transfer design elements from the design content and design assets to the selected design template to generate one or more designs. The functions of these modules will be described in detail later.

[0101] The server 110 may also include a data storage application 118 configured to receive and process requests to persistently store data related to operations performed / services provided by the server application 112 to and retrieve data from the data storage device 119. Such requests may be received from the server application 112, other server environment applications, and / or (in some cases) directly from a client application (e.g., 142).

[0102] The data storage application 118 may be, for example, a relational database management application or an alternative application for storing and retrieving data from the data storage device 119. The data storage device 119 may be any suitable data storage device (or set of devices), such as one or more non-transitory computer-readable storage devices, such as a hard disk, a solid-state drive, a tape drive, or an alternative computer-readable storage device. In addition, although a single instance of the data storage device 119 is described, the server system 110 may include multiple instances of the data storage device.

[0103] The data storage device 119 stores data related to the operations performed / services provided by the server application 112. Specifically, it stores asset libraries, such as a template library 122 (e.g., a library of design template records for various design categories), a font library 126 (e.g., a library of fonts and font palettes), a color library 124 (e.g., a library of colors and color palettes), and a media library 120 (e.g., a library of media items such as images and videos), usage metrics (e.g., Table D), user account data, design data (e.g., as shown in Tables A and B), and / or other data related to the operation of the server application 112. In addition, the data storage device 119 may also maintain a cache that stores design plan descriptors 128 as candidate designs are generated. Once a candidate design is generated and provided to a client device, the design plan descriptors may be discarded and the cache may be cleared.

[0104] The ML system 130 hosts one or more generative ML models that can be configured to generate outputs based on input prompts. Specifically, the ML system 130 can be configured to receive a first input prompt (e.g., a design outline prompt) and generate a design outline based on the first input prompt. The ML system also receives a second input prompt (e.g., a design content prompt) and is trained to generate design content based on the second input prompt.

[0105] In some embodiments, the ML system 130 may be a large language model (LLM) that is trained as a general ML model that can be used to generate different types of text-based outputs. In this example, if a general ML model is used, it is additionally trained to perform a specific task. For example, the general ML model may be trained to generate text based on a prompt (e.g., the design outline and design content described above). In other embodiments, the ML model may be a more specific model that is trained to generate the outputs described above.

[0106] Furthermore, in some examples, ML system 130 may be associated with and owned by the same party that operates server system 110. In this case, ML system 130 may be part of server system 110. In other examples, ML system 130 may be owned or operated by a third party independent of the party that owns or operates server system 110. Examples of third-party LLMs include OpenAI's ChatGPT4 and Google's Bard.

[0107] The client system 140 may be a desktop computer, a laptop computer, a tablet computing device, a mobile / smartphone, or other suitable computer processing system. The client system 140 hosts a client application 142, which, when executed by the client system 140, configures the client system to provide client-side functionality / interact with the server system 110 (or more specifically, the server application 112 and / or other applications provided by the server system 110). Through the client application 142 and as discussed in detail below, a user may access and use the various techniques and features described herein, for example, a user may enter a prompt to generate a design and view or preview the design. The client application 142 may also provide the user with access to additional design-related operations, such as creating, editing, saving, publishing, sharing, and / or other design-related operations.

[0108] The client application 142 may be a general purpose web browser application that accesses the server application 112 via an appropriate uniform resource locator (URL) and communicates with the server application 112 via a general purpose World Wide Web protocol (e.g., http, https, ftp) over the communication network 150. Alternatively, the client application 142 may be a local application that is programmed to communicate with the server application 112 using defined API calls and responses. A given client system such as 130 may have more than one client application 142 installed and executing thereon. For example, the client system 140 may have one or more general purpose web browser applications and local client applications.

[0109] This disclosure describes various operations performed by server application 102 and client application 142. However, operations described as being performed by a particular application (e.g., server application 112) may be performed by (or in conjunction with) one or more alternative applications (e.g., client application 142), and / or operations described as being performed by multiple separate applications may, in some cases, be performed by a single application.

[0110] In this example, server system 110 is configured to perform the functions described herein by executing a software application (or set of software applications), i.e., computer-readable instructions stored in a storage device (such as non-transitory memory 210 described below) and executed by a processing unit (such as processing unit 202 described below) of system 200. Similarly, client system 140 is configured to perform the functions described herein by executing software application 142 stored in a storage device and executed by a processing unit of the corresponding system.

[0111] The techniques and operations described herein are performed by one or more computer processing systems. For example, client system 140 can be any computer processing system configured (or configurable) by hardware and / or software (e.g., client application 142) to provide client-side functionality. Client system 140 can be a desktop computer, laptop computer, tablet computing device, mobile / smartphone, or other appropriate computer processing system. Similarly, server application 112 is also executed by one or more computer processing systems (server system 110).

[0112] Figure 2 A block diagram of a computer processing system 200 that may be configured to implement the embodiments and / or features described herein is provided. For example, Figure 1 The system 110 and / or 200 may be (or include) for example Figure 2 A computer processing system is shown (although alternative architectures are possible).

[0113] System 200 is a general purpose computer processing system. It should be understood that Figure 2 Not all functional or physical components of a computer processing system are shown. For example, no power supply or power supply interface is depicted, but system 200 carries a power supply or is configured to connect to a power supply (or both). It will also be appreciated that the particular type of computer processing system will determine the appropriate hardware and architecture, and that alternative computer processing systems suitable for implementing features of the present disclosure may have more, alternative, or fewer components than those described.

[0114] Computer processing system 200 includes at least one processing unit 202. Processing unit 202 may be a single computer processing device (e.g., a central processing unit, a graphics processing unit, or other computing device), or may include multiple computer processing devices. In some cases, when computer processing system 200 is described as performing an operation or function, all processing required to perform the operation or function will be performed by processing unit 202. In other cases, the processing required to perform the operation or function may also be performed by a remote processing device that is accessible and available (in a shared or dedicated manner) to system 200.

[0115] Through communication bus 204, processing unit 202 communicates data with one or more machine-readable storage (memory) devices that store computer-readable instructions and / or data executed by processing unit 202 to control the operation of processing system 200. In this example, system 200 includes system memory 206 (e.g., BIOS), volatile memory 208 (e.g., random access memory, such as one or more DRAM modules), and non-transitory memory 210 (e.g., one or more hard disks or solid-state drives).

[0116] The system 200 also includes one or more interfaces, generally indicated at 212, via which the system 200 is connected to various devices and / or networks. Other devices may be integrated with the system 200 or may be separate. Where a device is separate from the system 200, the connection between the device and the system 200 may be via wired or wireless hardware and communication protocols, and may be a direct or indirect (e.g., networking) connection.

[0117] Depending on the particular system in question, the devices to which system 200 is connected include one or more input devices that allow data to be input into / received by system 200, and one or more output devices that allow data to be output by system 200. Example devices are described below, but it should be understood that not all computer processing systems will include all of the devices mentioned, and additional and alternative devices to those mentioned may well be used.

[0118] For example, the system 200 may include or be connected to one or more input devices through which information / data is input into (received by) the system 200. For example, such input devices may include a keyboard, a pointing device (e.g., a mouse or a touch pad), a touch screen, and / or other input devices. The system 200 may also include or be connected to one or more output devices controlled by the system 200 to output information. For example, such output devices may include one or more display devices (e.g., an LCD, LED, a touch screen, or other display device) and / or other output devices. The system 200 may also include or be connected to devices that act as input and output devices, such as a touch screen display (which can receive touch signals / inputs and display / output data) and a memory device (from which data can be read and to which data can be written).

[0119] For example, where system 200 is an end-user device (e.g., system 140), it may include a display 218 (which may be a touch screen display), a camera device 220, a microphone device 222 (which may be integrated with the camera device), a cursor control device 224 (e.g., a mouse, touchpad, or other cursor control device), a keyboard 226, and a speaker device 228.

[0120] System 200 also includes a method for communicating with a network (e.g., Figure 1 The system 200 may include one or more communication interfaces 216 for communicating with the network 150. Via the communication interfaces 216, the system 200 may transmit data to and receive data from networked systems and / or devices.

[0121] The system 200 may be any suitable computer processing system, such as a server computer system, a desktop computer, a laptop computer, a netbook computer, a tablet computing device, a mobile / smartphone, a personal digital assistant, or an alternative computer processing system.

[0122] System 200 stores or has access to computer applications (which may also be referred to as computer software or computer programs). Such applications contain computer-readable instructions and data that, when executed by processing unit 202, configure system 200 to receive, process, and output data. The instructions and data may be stored on a non-transitory machine-readable medium, such as 210, accessible to system 200. The instructions and data may be transmitted to / received by system 200 through an interface (e.g., communication interface 216) via data signals in a transmission channel enabled by, for example, a wired or wireless network connection.

[0123] Typically, one application accessible to system 200 will be an operating system application. In addition, system 200 will store or access applications that, when executed by processing unit 202, will configure system 200 to perform the various computer-implemented processing operations described herein. For example, and with reference to the above Figure 1 In a networked environment, the server system 110 includes one or more systems 200 running server applications 112. Similarly, the client system 140 runs a client application 142. The application 142 is configured to display an input user interface (UI), for example, on a display 218. The server application 112 can communicate with the client application 142 to display the input UI using the user interface module 134. The input UI provides a mechanism for the user to, for example, generate designs by searching and selecting template designs, edit and add content to these template designs, and output designs by saving, publishing, or otherwise outputting designs. For example, a user can start generating a design by entering a search query for a design template (e.g., "welcome poster"), where the UI will present the user with various design templates suitable for poster category designs (e.g., poster type design category templates). The user can then view the templates, select a template, start editing a design from a template, and publish a design completed from a template. For example, the metric model 103 can save such searches, selections, publications, actions, and events as usage metrics.

[0124] In the present disclosure, an input UI provides a mechanism for a user to automatically generate a design by inputting a design prompt, and to edit and output such a design. Various input UIs are possible. One example is a graphical user interface (GUI), and in the following description, the UI will be envisioned as a GUI. Although a GUI is provided as an example, an alternative input UI may also be used. As another example, the input UI may be a command line interface type UI, which a user may use to provide prompts, media items and media item identifiers (e.g., file locations or other identifiers), and selections used in design generation. The UI also allows the user to access and enable the execution of other functions described herein. For example, the UI may include a prompt input area, which a user may use to input prompts. The UI may also include a function of displaying design options (e.g., displaying candidates) to the user for selection, which will be further explained below.

[0125] Example Method

[0126] Steering Figure 3 , a method 300 for automatically generating one or more candidate designs will be described. The operations of the method 300 will generally be described as being performed by the application 112 (and various associated modules) running on the system 110. However, the operations may be performed by one or more alternative applications running on the system 110 and / or one or more alternative computer processing systems.

[0127] Application 112 may be configured to execute method 300 in response to detecting one or more triggering events. As an example, application 112 may communicate with application 142 (eg, via network 150) to cause application 142 to display a user interface, such as Fig. 9 900. UI 900 includes a prompt input area 902. Prompt input area 902 may include a text field with placeholder text, such as "Use 5 or more words to describe your design" or alternative text, which instructs the user to enter their prompt in this area 902. The UI also includes interactive controls, such as a "generate design" control 904. When a user enters a user prompt in prompt input area 902 and then activates interactive control 904, method 300 may begin.

[0128] At 302, a request to generate a design is received at the server application 112. In one example, upon user activation of the control 904, the client application 142 creates a request to generate one or more candidate designs and passes a user prompt along with the request to the server application 112. The user prompt may be in the form of a text string, such as 5 or more words.

[0129] At 304, the application 112 creates a design plan descriptor 128 for the received request. Specifically, the application may generate a unique design plan identifier and store the design plan identifier in the design plan descriptor 128 along with the user input received as part of the request. Table F below shows an example design plan descriptor 128 at this stage:

[0130] Design plan identifier 273278 User Tips "Austria Welcome Poster"

[0131] Table F - Example Design Plan Descriptor after Step 304

[0132] At 306, category determination module 113 determines a category for the candidate design based on the input prompt. Category determination module 113 may store a list of design categories in data storage 119. In some examples, the user prompt may explicitly mention a design category. In such examples, category determination module 113 may identify a design category by parsing the input prompt and comparing terms from the input prompt to the list of design categories to determine a match.

[0133] In other examples, the user prompt may not explicitly mention a design category. In such cases, the category determination module 113 may utilize different techniques to identify a design category. In one example, the category determination module may do so based on usage metrics (e.g., as shown in Table E). For example, the metric category determination module 113 may look up usage metrics (e.g., as shown in Table E) for search queries containing the term "posts" and determine which category of designs accounts for the largest percentage of published designs for that term.

[0134] In other examples, the category determination module 113 may include or utilize an ML model that is trained to identify a design category given an input prompt. In this case, the machine learning model may be trained on a large number of input prompts or search terms labeled with appropriate design categories. After sufficient learning, the ML model is trained to generate a design category (e.g., from a set of design categories) for any given input prompt with reasonable accuracy.

[0135] The category determination module 104 may combine various category determination techniques to determine the category of the input design. For example, the category determination module may apply a weighted relevance score of the design category to the input terms based on the number of matches of words in the input to words in the design category. The relevance of the design category to the input terms may also be assigned (and / or weighted) based on the proportion of design posts in the category caused by the corresponding input terms. Although the present example is about a single category, in alternative embodiments, the category determination module may determine multiple categories or a range of categories relevant to the input prompt, and may determine multiple categories or a distribution of categories relevant to the design.

[0136] Once the category determination module 113 determines one or more categories of the design, it updates the design plan descriptor 128. Table G below shows an example of a design plan descriptor at this stage:

[0137] Design plan identifier 273278 User Tips "Austria Welcome Poster" category poster

[0138] Table G - Example Design Plan Descriptor after Step 306

[0139] At 308, the application 112 generates a design outline for one or more candidate designs. In some embodiments, this includes transmitting a user prompt to the ML system 130 along with a request to generate a design outline. The application 112 receives the design outline from the ML system 130 and populates the design plan descriptor with the received design outline. As previously described, the design outline includes a style descriptor and a media query.

[0140] Will refer to Figure 4 Method step 308 is described in more detail. An example design plan descriptor 128 at this stage is shown in Table H below:

[0141] Design plan identifier 273278 User Tips "Austria Welcome Poster" category poster Style Descriptor Style Descriptors: Mountain landscapes and stunning natural scenery Media Inquiries Media Enquiries: The majestic Austrian Alps

[0142] Table H - Example Design Plan Descriptor after Step 306

[0143] At 310, once the design outline is generated, the application 112 generates, selects, or retrieves design components for the candidate design. For example, this may be accomplished by selecting one or more design templates for the candidate design and selecting design assets (e.g., media items and style attributes) based on the design outline generated from one or more asset libraries at step 304. Figure 5 Describe this step in more detail.

[0144] At step 312, application 112 generates text content for the design based on the design outline and the selected design template. In some embodiments, to this end, application 112 generates a second input prompt based on the design outline and the design assets, and transmits this prompt together with the request for the design text to ML system 130. Application 112 receives the text content from ML system 130 and fills the design plan descriptor, in particular the text content field, with the received text content. Figure 6 Method step 312 is described in more detail.

[0145] Table A shows an example design plan descriptor 128 at this stage.

[0146] The method 300 may then proceed to step 314, where one or more candidate designs are generated and transmitted to the client system 140 for display thereon. At step 314, the application 112 examines the design plan descriptor 128 and transmits design elements such as text content and media items, and applies style attributes such as fonts and colors to the selected design template to create a candidate design record. Figure 7 These steps are further summarized. Although the method 300 is described in a sequential order, the steps of the process may be performed in an alternate order.

[0147] Figure 4 An example method 400 for performing method step 306, i.e., generating a design outline, according to an embodiment of the present disclosure is shown. Method 400 begins at step 402, where the user input prompt received at step 302 is provided to prompt generation module 115, and a design outline prompt is generated. The content of the design outline prompt will depend on the type of ML system 130 used. If ML system 130 is a general purpose LLM, the design outline prompt contains the user input prompt and configuration data, which provides instructions to ML system 130 to generate a page outline. In the case where ML system 130 includes a specific ML model that has been trained for the specific task of generating a design outline, the design outline prompt may only contain the user input prompt.

[0148] The precise format of the configuration data depends on a variety of factors, including the type of LLM (e.g., the configuration data used with OpenAI’s ChatGPT may differ from the configuration data required for Google’s Bard), the training mechanism for the ML model, and the content of the user input prompt (and / or other available data).

[0149] In one example, the configuration data for the page outline prompt may include a brief description of the task (e.g., a design outline for generating a design), parameters of the task (e.g., output format, number of elements, output tone, rules, etc.), and input prompts and one or more training examples of design outlines that the ML system 130 is expected to generate based on these input prompts. Table I below shows an example of design outline configuration data that can be used.

[0150]

[0151] Table I - Example Design Outline Configuration Data

[0152] It will be appreciated that the configuration data may contain many alternative components and template hints, and that many alternative methods of generating hints are possible. For example, a parameter component may specify the generation of multiple design outlines, and / or each design outline may contain multiple style descriptors and / or multiple media queries. In one example, the configuration data may be (or contain) a single pre-assembled template hint, such as a string containing all relevant set text components. Alternatively, separate hints may be generated containing individual components and combinations thereof. Thus, the ML system 130 may be configured by providing the configuration data as a hint, a portion of a hint, or a series of hints.

[0153] In some embodiments, the same configuration data may be used each time to configure the ML system 130 to generate a page outline. In this case, the design outline configuration data may be predefined and stored in the data storage device 119. In other embodiments, the configuration data may vary (e.g., depending on user requirements). For example, a user may provide a tone for the design (e.g., funny) that overrides a preset tone for the design in the configuration data. In this case, the parameters of the configuration data may be updated to include the user input tone before the configuration data is added to the design outline prompt.

[0154] At step 402, the prompt generation module 115 retrieves the configuration data from the data storage device 119, determines whether the configuration data needs to be updated (e.g., based on the received user input), and combines the configuration data with the user input prompt to generate a page outline prompt. In one embodiment, the prompt generation module 115 generates the design outline prompt by constructing a text string from one or more component parts of the configuration data and the user input prompt (e.g., by concatenating the component parts and the user input prompt).

[0155] At step 404, once the design outline hints are generated, the hint generation module 115 transmits the design outline hints to the ML module 114. The ML module 114 then transmits the design outline hints to the ML system 130.

[0156] Through the design outline configuration data, the ML system 130 is prompted to generate a design outline based in part on the user prompts. Based on the example configuration data shown in Table I, the ML system 130 is prompted to generate a style descriptor and a media query. The style descriptor can be a text string that describes the thematic style of the design based in part on the user prompts. The media query can be a string that describes suitable media items for the design based in part on the user prompts.

[0157] At 406, the ML module 114 receives the design outline output by the ML system 130 as an output text string, which is referred to as completion. An example of the output generated by the ML system 130 at this time for the user input "Austria Welcome Poster" is provided in the following Table J in the format of "Style Descriptor": {} and "Media Query": {} -

[0158]

[0159] Table J: Example Design Outline

[0160] In the above examples, a single instance of a style descriptor and a media query is provided. In alternative embodiments, a design outline may contain multiple style descriptors and / or multiple media queries. By defining the requirements in the configuration data parameters, multiple style descriptors and / or multiple media queries may be appropriately generated. Additionally or alternatively, the same user prompts and design outline configuration prompts may be provided to multiple LLMs (or instances of LLMs) with different parameters (e.g., different temperature settings, training sets, or a few shot examples) to generate multiple design outlines with differences in their corresponding style descriptors and / or media queries. As will be further outlined below, such style descriptors and media queries are used for retrieval and generation of design components. Thus, changing style descriptors and media queries in a controlled manner can make the design components more diverse while maintaining alignment with the original input prompts.

[0161] At step 408, the design outline is added to the design plan descriptor 128. In some embodiments, the design generation module 116 may parse or process the completed text based on the formatting rules specified in the configuration data to identify style descriptors and media queries. For example, the design generation module may parse the completed text and identify the term "style descriptor" or "media query" and then identify the string after the colon (":") until the carriage return. Alternative parsing, text analysis and processing techniques may also be used to identify style descriptors and / or media queries in the design outline.

[0162] Once the individual components of the design outline are determined, the design generation module 116 stores the style descriptors and media queries in the design plan descriptor 128 as described above. The method 400 then ends.

[0163] It should be appreciated that in method 400, it is assumed that configuration data is provided to the ML system 130 each time a new page outline is required. However, this is not necessarily required in all embodiments. In other embodiments, design outline configuration data may be provided to the ML system 130 each time an instance of the ML system 130 is invoked. If the same ML system instance is subsequently used for subsequent design outline requests, the configuration data does not need to be submitted to the ML system 130 again, because the ML system can remember the configuration data it previously provided and use the configuration data for subsequent design outline requests. Once an instance of the ML system 130 is closed or exited, the instance can refresh the configuration data, and the server application 112 may need to resend the configuration data along with the page outline request when a new instance of the ML system 130 is invoked.

[0164] Furthermore, in method 400, it is assumed that the ML system is a general purpose LLM that has not been previously trained or configured to provide page outlines in the desired manner. However, this is not necessarily the case in all embodiments. In some embodiments, a dedicated ML system may be employed that has been trained and / or fine-tuned using a large amount of training data of input user prompts and desired output design outlines. There is no need to provide additional configuration data for such a specially trained ML system 130, and in this case, the design outline prompts may simply contain the user prompts.

[0165] Figure 5 An example method 500 for performing method step 310 (ie, retrieving design components) is shown.

[0166] The method 500 begins at step 502 , where the design generation module 116 retrieves style descriptors and / or media queries from the design plan descriptor 128 .

[0167] At step 504, the design generation module 116 selects one or more style palettes for the design. The selection of the style palette is based on the retrieved style descriptors. In some embodiments, the style palette includes one or more color palettes and one or more font palettes. For example, a color palette may be selected by performing a search in the "Tag" metadata field in the color palette library using one or more style descriptors as search terms. Similarly, a font palette may be selected, for example, by performing a search in the "Tag" metadata field in the font library using the style descriptor as a search term. It should be understood that multiple color or font palettes may be associated with the same tag, and therefore the design generation module 116 may select more than one color palette and / or font palette for the style descriptor at this stage. In some embodiments, only a color palette is selected or only a font palette is selected. The number (and type) of style palettes selected may depend on how the design generation module 116 is configured.

[0168] Furthermore, where a design plan descriptor contains multiple style descriptors, and a search is performed for a style palette for each of the style descriptors, respectively, the same style palette may be selected for different style descriptors (e.g., because the style palette is associated with multiple tags). In some cases, this may not be a problem, and the same style palette may be selected for different style descriptors. In other cases, if it is desired to create multiple designs with many different design styles based on the input prompt, the design generation module 116 may be configured to discard the duplicate style palette and replace it with another style palette that substantially matches the corresponding style descriptor.

[0169] At step 506, the design generation module 116 retrieves one or more design templates based on the design category and the style descriptor. In some embodiments, the design generation module 116 may perform a search in the 'design type' field in the design template records stored in the template library 122 using the design category identified at step 306 to obtain a first subset of results. The design generation module may then perform a search in the style field of the template records in this subset of results using the style descriptor to identify one or more design templates for the one or more candidate designs. In other embodiments, the design generation module 116 may formulate a search string combining the style descriptor and the design category, for example based on the presence of the search terms in the template metadata, and use this search string to identify relevant design templates.

[0170] It should be understood that multiple design templates may match a style descriptor and / or a category type. In this case, the design generation module 116 may be configured to select a predetermined number (e.g., 1, 2, 5, etc.) of design templates from the matching results. In addition, if the design plan descriptor contains multiple style descriptors, the design template may select one or more design templates for each style descriptor.

[0171] Furthermore, where the design plan descriptor contains multiple style descriptors, and a search is performed for a design template for each of the style descriptors separately, the same design template may be selected for different style descriptors (e.g., because the design template is associated with multiple style categories). In some cases, this may not be a problem, and the same design template may be selected for different style descriptors. In other cases, if it is desired to create multiple designs with many different design styles based on the input prompt, the design generation module 116 may be configured to discard the duplicate design template and replace it with another design template that substantially matches the corresponding style descriptor.

[0172] At step 508, the design generation module 116 generates analysis data based on the selected design templates. Generating the analysis data includes retrieving (e.g., from the template library 122) additional metadata about the selected design templates and analyzing the corresponding metadata. For example, the design generation module 116 may analyze the template metadata to determine the presence and type of various placeholder elements (e.g., placeholder text elements and placeholder shape elements) included in each design template.

[0173] The analysis data can be hierarchical, that is, each element of each template can be regarded as an analysis object and analyzed to determine the analysis data of each element. For example, each element of the template can be analyzed to determine whether it is a text element or a shape element. The shape element can be analyzed to determine whether it is configured to receive media items, color fill or alternative fill type content. For example, the target media item element can be analyzed to be configured to receive images and / or videos. The text element can be analyzed to determine the text hierarchy level, such as level 0, level 1, level 2 or level 3, and / or as one of a leading title, title, subtitle or body. The analysis data of all design elements of the template can be aggregated to generate the analysis data of the corresponding template.

[0174] The analysis data can be determined by processing and identifying attributes that are explicitly defined in the metadata of the corresponding element. For example, a placeholder shape element of a template may have an element record that explicitly defines that its expected fill type is an image type fill. Therefore, the design generation module 116 can generate analysis data for such an element as an image type placeholder element. Similarly, depending on the specific design template, the placeholder text element record can explicitly define the corresponding text hierarchy level. Alternatively, the design generation module 116 analyzes the text elements of the template to determine the text hierarchy level. Such a determination can be based on one or more of the following, for example: a default font size defined in the element record of the text element; a relative font size between the font sizes defined for the text elements in the template; the (x, y) position of the element, the relative (x, y) position of the element, a specific term in the element placeholder text; and other factors. As an example, a text element with the placeholder text "Put the leading title here" or "Please put your subject here" can be determined as a text element with a text hierarchy level of a title, and a text element with an x ​​coordinate higher than the title element can be determined as a text element with a text hierarchy level of a leading title. It will be appreciated that many alternative processes for determining analytical data are possible.

[0175] At step 510, the design generation module 116 selects one or more media items from the media library based on the media query present in the design plan descriptor. To this end, the design generation module 116 searches the media library using the media query to identify one or more media items that match the media query. In one example, the design generation module 116 can pass the media query to the data storage application 118, which searches the media library 120 for one or more media items that match the media query. The data storage application 118 transmits the identified media items to the design generation module 116.

[0176] The design generation module 116 then selects a predetermined number of media items from the results. The predetermined number may be preset and based on the configuration of the design generation module 116. In some examples, the number of media items selected may depend on the number of design templates selected at step 506 and the number of image placeholder elements detected in the selected design templates at step 508.

[0177] In one example, the design generation module 116 selects the first n media items in the result list (because they are likely to have the highest percentage of matches to the media query). In other examples, the design generation module 116 may randomly select n media items from the result list (if all media items are equally matched to the media query). In some examples, the design generation module 116 may select n media items from media items having sizes and / or aspect ratios within a predetermined range, e.g., media items that are generally compatible with templates of the determined one or more design categories. Additionally or alternatively, the media items may be selected based on a weight of relevance (e.g., order in a list of search results returned) and their corresponding sizes and / or aspect ratios.

[0178] At 510, the design elements (e.g., media items), style attributes (e.g., color and font palettes), design templates retrieved at steps 502-506 and 510, and the analytical data determined at step 508 are stored in the design plan descriptor 128. In some embodiments, the design generation module 116 may store the design asset identifiers and analytical data in a design component field in the design plan descriptor. In other examples, separate design components may be stored in separate fields, such as color palettes, font palettes, media items, design templates, template analytical data, etc.

[0179] Although method 500 is described sequentially, this is not necessarily true for all embodiments. In some cases, method steps 504, 506, and 508 may be performed substantially simultaneously.

[0180] Figure 6 An example method 600 for performing step 312 (i.e., generating text content) is shown. In some embodiments, the text content can be generated as replacement text for a placeholder text element, for example, in each design template in a design plan descriptor. To this end, the method 600 begins at step 602, where the design generation module 116 retrieves design template analysis data from the design plan descriptor. Specifically, the design generation module can retrieve analysis data corresponding to the placeholder text element of each design template in the design plan descriptor.

[0181] At step 604, the design generation module 116 passes the retrieved design template analysis data along with the input prompt to the prompt generation module 115. The prompt generation module 115 generates one or more text content prompts based on the received data.

[0182] In some examples, the text content prompt includes configuration data and prompt data. The configuration data may include a brief description of the task (e.g., generating a designed text content), parameters of the task (e.g., output format, output tone, rules, etc.), and one or more training examples of input prompts and text content that the LLM expects to generate based on the input prompts.

[0183] Table K below shows an example of design outline configuration data that may be used.

[0184]

[0185]

[0186] Table K - Example Design Outline Configuration Data

[0187] It should be appreciated that the configuration data may include many alternative components. For example, the configuration data may be (or include) a single preassembled template prompt, such as a string including all relevant set text components.

[0188] In one example, the parameters of the configuration data may be different for different types of placeholder text elements. For example, if one placeholder text element is of the "title" type and another placeholder text element is of the "body" type, the specific text format required for each element type may be different. Therefore, in some embodiments, the prompt generation module 115 identifies the placeholder text element type of each design template from the analysis data and updates the configuration data so that the parameter portion of the configuration data matches the placeholder text element type of the design template. For example, if a given design template contains only title and body placeholder elements, the configuration data parameters can be updated to include instructions such as "The output format must be: "-Title: Title text content"; "-Body: Body content"". Alternatively, if a given design template contains leading title and subtitle placeholder text elements, the configuration data parameters can be updated to include instructions such as "The output format must be: "-Leading title: Leading title text content"; "-Subtitle: Subtitle text content"".

[0189] Furthermore, since the length and type of text content may be different for different types of placeholder text elements, any number of words, characters, sentences, or bullet points may be appropriately specified in the configuration data. The type of the few shot examples provided may also vary depending on the type of placeholder text element present in the design template.

[0190] Returning to method step 504, once the prompt generation module 115 receives the design template analysis data, it identifies the placeholder text element type required for each design template (e.g., by examining the corresponding analysis data), and then updates the parameters of the configuration data based on the identified placeholder text element type.

[0191] Then, the prompt generation module 115 generates design content prompts for each design template in the design plan descriptor. To this end, for each design content prompt, the prompt generation module combines the user input prompt with the updated configuration data.

[0192] At step 606, once the design content hints are generated, the hint generation module 115 transmits the design content hints to the ML module 114. The ML module 114 then transmits the page content hints to the ML system 130. In some embodiments, where multiple design content hints are created, the design content hints may be transmitted simultaneously to the ML system 130. In other embodiments, the design content hints may be transmitted sequentially, such as after receiving the output of the ML system 130 for a previous design content hint.

[0193] Through the text content configuration data, the LLM is prompted to generate text content based in part on the user prompts and the configuration data parameters of each design template. For example, based on the example configuration data shown in Table K, the ML system 130 can be prompted to generate different types of text content (e.g., leading titles, subtitles, titles, body text, etc.) of different structures and text lengths. The ML system 130 outputs the text content according to the corresponding design content prompts.

[0194] At 608, ML module 114 receives the design content output by ML system 130 for each design content prompt as an output text string, which is referred to as completion. ML module 114 passes the output to design generation module 116, which updates the design plan descriptor with the received text content.

[0195] At step 610, the design generation module 116 processes the completions, which may include analyzing each completion to identify the expected corresponding text elements based on the template analysis data. For example, the design generation module 116 may parse or process the output text to identify a text string that appears after one or more of the following as defined by the output format of the text in the configuration data: "-Leading Title:"; "-Title:"; or "-Subtitle:". Additionally or alternatively, the text content may be identified based on line feeds, carriage returns, and / or special characters that may be defined in the configuration data. Many alternative parsing, text analysis, and processing techniques may also identify text elements in a completion.

[0196] In step 612, the design generation module 116 stores the text elements of the identification of each design template in the design plan descriptor. For example, the text elements can be stored as text element records with text content and text hierarchy level respectively. The text elements can be mapped to the template analysis and placeholder text therein for generating the template ID of the corresponding design template of the corresponding text element. In certain embodiments, the text elements can be stored in the design plan descriptor as the corresponding text content data (and metadata) of the corresponding design template. Then, method 600 ends.

[0197] In one example, for an input prompt "Magic design software architecture page social media post" associated with four design templates (e.g., Template 1; Template 2; Template 3; Template 4), four design content prompts may be generated by combining the user prompt and the updated configuration data (as shown in 604). The text content prompt is then passed to the ML system 130 (as shown in 606), which generates text content for each template.

[0198] The following table L shows the output text of the ML system for 4 content prompts −

[0199]

[0200] Table L: Example text content output

[0201] While it is contemplated that the design plan may be generated by providing separate corresponding design content prompts (eg, input prompts and updated configuration data) to the ML system 130, in some embodiments, a single design content prompt may be generated for all design templates.

[0202] It should be appreciated that in method 600, it is assumed that configuration data is provided to the ML system 130 each time new page content is required. However, this is not necessarily required in all embodiments. In other embodiments, page content configuration data for all different page types may be provided to the ML system 130 each time an instance of the ML system 130 is invoked. If the same ML system instance is subsequently used for subsequent page content requests, the configuration data does not need to be submitted to the ML system 130 again, because the ML system 130 can remember the configuration data it previously provided and use the configuration data for subsequent page content requests. Once the ML system instance is closed or exited, the instance may refresh the configuration data, and when a new instance of the ML system 130 is invoked, the server application 112 may need to resend the configuration data for all page types and the page outline request. In addition, while the page content is described as being generated by providing separate corresponding page content prompts (e.g., user input prompts and corresponding page titles and page configuration data) to the ML system 130, in some embodiments, a single page content prompt may be generated for all pages or all pages of the same page type.

[0203] Furthermore, it is assumed that the ML system 130 is a general purpose LLM that has not been previously trained or configured to provide page content in the desired manner. However, this is not necessarily the case for all embodiments. In some embodiments, a dedicated ML system may be employed that has been trained using a large amount of training data for page types, titles, and user prompts, as well as the desired output page content. There is no need to provide additional configuration data for such a specially trained ML system 130, and in this case, the page content prompt may simply contain the user prompt and the page title.

[0204] Furthermore, while methods 400 and 600 contemplate separate instances of ML system 130 generating the design outline and textual content for each design template, in some embodiments, a single instance of ML system 130 may generate both the design outline and the textual content.

[0205] Reference now Figure 7 , describes a method 700 for generating and outputting a final design. The method 700 will be described with reference to generating and outputting a plurality of candidate designs for final selection by a user.

[0206] At 702, the design generation module 116 retrieves the design template identified in the design plan descriptor 128. For example, where the design plan descriptor includes an identifier of the selected design template, the design generation module 116 performs a search in the template library using the template identifier and retrieves the corresponding design template record.

[0207] At 704, for each design template record, the text content from the design plan descriptor is transferred to the corresponding placeholder text element field. For example, for the first template (Template 1), in which the text content {Leading Title 1: "The Magic of Design"}, {Title 1: "Discover the Best Architecture Design Software"} is generated, the corresponding leading title and title placeholder text element fields can be updated with the text "The Magic of Design" and "Discover the Best Architecture Design Software", respectively. Similarly, other placeholder text elements of other design templates are also updated with the corresponding text in the design plan descriptor.

[0208] At 706, for each design template record having a media placeholder element, a media item is selected from the design plan descriptor and the media placeholder element field in the design template record is updated with the selected media item identifier. In some embodiments, the media item is selected for a particular design template based on the size and / or aspect ratio of the placeholder element and the size and / or aspect ratio of the available media items in the design plan descriptor. If necessary, the media item can be cropped and / or scaled to fit the corresponding template placeholder element.

[0209] At 708, the color palette is applied to the design template record. This may include the design generation module 116 retrieving the palette identifier from the design plan descriptor and adding it to the color palette field of the corresponding design template record. The application of the color palette actually involves recoloring the text and shape elements in the template according to their default colors. For example, such recoloring can be performed according to a method such as method 500 described in Australian patent application 2021273643, entitled "Systems and methods for automatically recolouring a design" filed on November 26, 2021. In this case, the populated template and the color palette associated with the populated template can be used as the input color palette and input design of method 500 of Australian patent application 2021273643.

[0210] At 710, a font palette is applied to a design template record. This may include the design generation module 116 retrieving a font palette identifier from a design plan descriptor and adding it to a font palette field of the corresponding design template record. Application of the font palette may be performed based on associating the fonts in the font palette with corresponding text hierarchy levels. For example, a font palette may include a title type font that is intended to be bold, concise, and eye-catching; and a body type font that is intended to be neat, clear, and easy to read. For each font palette and template, the corresponding font is then formatted for text displayed in each template. The font palette may also format for font size, centering, or alignment, and format text as bold, underlined, italic, etc.

[0211] In steps 706, 708, and 710, the design generation module 116 may randomly select corresponding media items, font palettes, and color palettes for the design template; according to the order in which the corresponding templates and media items, color palettes, and font palettes are retrieved from the corresponding libraries; and / or according to predefined criteria, such as most compatible, most relevant, etc. For example, the relevance of the media items to the template may be determined based on a comparison of matching terms (or synonyms) in the media item metadata (e.g., image descriptors or tags) with terms in the title text content of the corresponding template. The relevance of the color palette and the font palette may also be determined based on matching terms (or synonyms) between the corresponding style palette metadata and the template text content. In addition, the relevance of the color palette may be selected based on key color data of the media items selected for the template. In some embodiments, additional or alternative colors or color palettes may be extracted from the media items selected for the template. In another example, an ML model may be trained on a training set of designs with text content and media, where the designs are ranked (e.g., by user ratings), and then the ML model is configured to select a combination of media items, font palettes, and color palettes for candidate designs.

[0212] At step 712, the design generation module 116 generates candidate design records for each updated template record. The design generation module 116 may save these candidate design records in the data storage device 119 (e.g., in a temporary cache) using unique identifiers. In addition, the design generation module 116 transmits the candidate design records to the client application 142. At this stage, the design plan descriptor 128 may be discarded.

[0213] The client application 142 may then use the candidate design records to present the candidate designs for display on the UI on the client device 140. For example, for each candidate design, the client application 142 may retrieve the design template mentioned in the corresponding candidate design record from the database, add the text components mentioned in the placeholder element field in the record to the corresponding placeholder element, retrieve the media items mentioned in the corresponding design record from the media library and fill them in the corresponding placeholder element, and apply the font palette and color palette mentioned in the design record.

[0214] Figure 8 An example UI 800 is shown that is generated and displayed on client system 140 at the conclusion of method 300. In this example, UI 800 displays four designs 802-808.

[0215] Once the candidate designs are displayed, editing functions may also be provided to the user within UI 800 to replace selected design assets between the candidate designs. For example, the user may be able to swap media items, color palettes, and font palettes between the displayed candidate designs. Other editing functions may include, for example: adding elements to a design; deleting elements from a design (including deleting elements that have been automatically generated for a design); editing elements that have been added to a design (including editing elements that have been automatically generated for a design); and / or other operations.

[0216] UI 800 may also provide the user with various options for exporting the output design. This may include, for example, one or more options that allow the user to: determine the export location (e.g., on a local memory such as 210 or a network accessible storage device); determine the export format (e.g., file type); determine the export size / resolution; and / or other export options.

[0217] The UI 800 may also provide the user with various options to share the design. This may include, for example, one or more options that allow the user to determine the format (e.g., file type) and then share the resulting design (e.g., by attaching the design to an electronic communication, uploading to a web server, uploading to a social media service, or sharing in other ways). The application 142 may also provide the user with the option of sending a link (e.g., a URL) to the design (e.g., by generating a link and attaching the link to the electronic communication or allowing the user to copy the link).

[0218] Other embodiments

[0219] In some examples, different text content may be generated and populated in each design template, and in alternative examples, the same text content may be used across the same text hierarchy level text elements of each design template. That is, all candidate designs may have the same leading title, title, subtitle, and / or body content.

[0220] In methods 300-700, application 112 generates one or more candidate designs. It should be appreciated that application 112 can be configured to generate any number of candidate designs using any number of style descriptors and media queries to retrieve any number of media items, color palettes, font palettes, and templates. In addition, not all selected media items (or style attributes) need to be used in the candidate designs. For example, six media items may be retrieved, but only four may be selected as candidate design records. In some examples, a user may specify the number of candidate designs when providing an input prompt. In this case, the user-specified number overrides any default settings of application 112.

[0221] Furthermore, although methods 300-700 describe selecting style attributes such as colors and fonts during the design process, this is not required in all embodiments. In some cases, the selected design template may include its own default color and font palette. In this case, colors and fonts may not be selected at step 504, and this step may be omitted.

[0222] Additionally or alternatively, a design component may be used for more than one candidate design. For example, the same media item may be displayed in two or more candidate designs. As another example, multiple candidate designs may utilize the same color palette.

[0223] Furthermore, although the candidate designs are described herein as single-page designs, the methods and systems described herein can be used to create multi-page designs.

[0224] The flowcharts shown in the figures and described above define operations for explaining various features in a particular order. In some cases, the operations described and illustrated may be performed in an order different from that shown / described, one or more operations may be combined into a single operation, a single operation may be divided into multiple separate operations, and / or the functions implemented by one or more described / illustrated operations may be implemented by one or more alternative operations. In addition, the functions / processing of a given flowchart operation may potentially be performed by (or in conjunction with) different applications running on the same or different computer processing systems.

[0225] The present disclosure provides various user interface examples. It should be appreciated that alternative user interfaces are possible. Such alternative user interfaces may provide user interface features that are the same or similar to the user interface features described and / or illustrated in a different manner, provide additional user interface features to the user interface described and / or illustrated, or omit certain user interface features that have been described and / or illustrated.

[0226] In the above embodiments, the operations of methods 300, 400, 500, 600, and 700 are described as being performed by application 112 (and various associated modules) running on a single computer processing system. However, these operations may be performed by one or more alternative applications running on system 110 and / or one or more alternative computer processing systems. For example, one or more of modules 113-117 may be different applications (running on the same or separate computer processing systems) that interoperate with application 112 to perform the described techniques. In another example, the functions performed by modules 113-117 may be combined in a design generation service that may be accessed by any suitable application (e.g., a web browser or other application). In another example, the functions of modules 113-117 may be provided by one or more client applications. In this case, application 142 may be configured to perform related operations for generating a design (e.g., the operations of modules 105-118).

[0227] As another example, the functions performed by modules 113-117 can be combined in a design generation package, which can be used to extend the functions provided by any design production application. In this case, the design generation package can be installed locally on a given end-user system, such as as a plug-in or extension to an existing video production application.

[0228] In the above description, some operations and features are explicitly described as optional. This should not be interpreted as indicating that if an operation or feature is not explicitly described as optional, it should be considered essential. Even if an operation or feature is not explicitly described as optional, it may still be optional.

[0229] Unless otherwise stated, the terms "include" and "comprise" (and variations such as "including", "includes", "comprising", "comprises", "comprised", etc.) are inclusive and do not exclude other features, components, integers, steps or elements.

[0230] In some cases, the present disclosure may use the terms "first", "second", etc. to describe various elements. Unless otherwise specified, these terms are only used to distinguish elements without order meaning. For example, without departing from the scope of the described examples, a first element or feature may be referred to as a second element or feature, and vice versa. In addition, when using terms such as "first", "second", etc. to distinguish elements or features rather than indicating an order, the second element or feature may exist without the first element or feature. For example, a second element or feature may appear before a first element or feature (or the first element or function has never appeared).

[0231] It should be understood that the embodiments disclosed and defined in this specification extend to alternative combinations of two or more of the individual features mentioned or evident from the text or drawings. All of these different combinations constitute alternative embodiments of the present disclosure.

[0232] This specification describes various embodiments with reference to many specific details, which may vary from implementation to implementation. Any limitation, element, attribute, feature, advantage or property not expressly recited in the claims should not be considered a required or essential feature. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0233] The background information described in this specification is the background information known to the inventor. Citing this information as background information does not mean to admit or imply that this background information is the prior art or common knowledge of ordinary technicians in the field.

Claims

1. A computer-implemented method for automatically generating a design, the method comprising: receiving an input prompt for generating the design; generating a design outline based on the input prompt, the design outline comprising a style descriptor for the design; Retrieving a design template based on the design outline, the design template comprising a placeholder design element; Retrieving at least one design element of the placeholder element; as well as The design is generated by populating the placeholder elements with the design elements.

2. The computer-implemented method of claim 1 , wherein generating the design outline comprises: generating a design outline prompt, wherein the design outline prompt includes the input prompt; and The design outline prompt is transmitted to a first machine learning system, and the first machine learning system is configured to generate the design outline based on the design outline prompt.

3. The method of claim 2, wherein the design outline prompt further comprises configuration data for configuring the first machine learning system to generate the design outline.

4. The computer-implemented method of any one of the preceding claims, wherein retrieving the design element comprises: generating a design element prompt, wherein the design element prompt includes the input prompt; The design element prompt is transmitted to a second machine learning system, the second machine learning system being configured to generate the design element based on the input prompt.

5. The computer-implemented method of claim 4, wherein the design element hint further comprises configuration data for configuring the second machine learning system to generate the design element, wherein the configuration data is based on a type of the placeholder element.

6. The computer-implemented method of claim 5, wherein generating the design element comprises: Data associated with the design template is processed to identify the placeholder element. 7 . The computer-implemented method of claim 6 , wherein the type of the placeholder element comprises a text hierarchy level, and the design elements are retrieved based on the text hierarchy level.

8. The computer-implemented method of any preceding claim, wherein the design outline comprises a media query.

9. The computer-implemented method of any one of the preceding claims, wherein retrieving the design template comprises searching a template library using the style descriptor and retrieving the design template from the search results.

10. The computer-implemented method of any preceding claim, further comprising searching a style property palette in a style property library using the style descriptor and selecting the style property palette.

11. The computer-implemented method of claim 10, wherein generating the design comprises applying the style attributes to the design template.

12. A computer-implemented method according to claim 10 or 11, wherein the style attributes include a color palette for coloring shape elements of the design; and a font palette for formatting text elements of the design.

13. The computer-implemented method of any one of claims 8 to 12, further comprising searching a media item library for a media item using the media query, and selecting the media item.

14. The computer-implemented method of claim 13, wherein the design template comprises a placeholder shape element, and the method further comprises populating the placeholder shape element with the media item.

15. A computer processing system comprising: one or more computer processing units; and A non-transitory computer readable medium storing instructions which, when executed by the one or more computer processing units, cause the one or more computer processing units to perform the method according to any one of claims 1 to 14.

16. A non-transitory storage device storing instructions, the instructions being executable by one or more computer processing units to cause the one or more computer processing units to perform the method according to any one of claims 1 to 14.

Images