System and method for maintaining state information when rendering a design interface in a simulation environment - Patents.com

Abstract

The computing system implements a simulation environment graphic design system. The graphic design system can be used to create a plurality of cards, each including a design element. The computing system can generate a production environment rendering of the individual cards as a simulation. In generating the production environment rendering of the simulation, the computing system processes each card in the sequence to determine a semantic structure of the card sequence, the semantic structure including nodes that represent the production elements of the simulated design. The computing device can further determine, based on the determined semantic structure, whether each design element of a first card and a second card in the sequence represents the same production element of the simulated design.

Description

Related Applications

[0001] This application claims the benefit of priority to (i) U.S. Patent Application No. 18 / 378,248, filed October 10, 2023, and (ii) U.S. Provisional Patent Application No. 63 / 414,900, filed October 10, 2022, the entire contents of each of which are incorporated herein by reference. [Technical Field]

[0002] The examples described herein relate to systems and methods for rendering a design interface in a simulation environment. [Background technology]

[0003] Software design tools come in many forms and uses. For example, in the field of application user interfaces, software design tools require designers to blend the functional aspects of a program with aesthetic and even legal requirements, resulting in a collection of pages that form the application's user interface. For a given application, designers often have many objectives and requirements that are difficult to keep track of. To assist designers in their work, some design tools allow for production environment simulations of cards (or arrangements of other design elements). For example, a production environment simulation can be implemented by rendering a sequence of cards to reflect state changes that may occur in the production environment. The use of such simulations allows designers to see how their design interfaces will be implemented in the production environment, enabling them to develop design interfaces with the production environment in mind. Summary of the Invention

[0004] According to an embodiment, a computer system generates a production environment rendering of a graphic design interface. In generating the production environment rendering, the computing system processes individual cards, each of which includes a set of design elements of a user interface or presentation to be simulated. The computing system processes the individual cards to determine a semantic structure of the sequence of cards, the semantic structure including nodes that represent the production elements of the simulated design. The computing device can further determine, based on the determined semantic structure, whether each design element of a first card and a second card in the sequence represents the same production element of the simulated design.

[0005] As described using various examples, the computing system operates to determine when design elements on different cards are semantically identical. Additionally, the computing system can operate to identify stateful design elements that represent production environment elements that undergo state changes (e.g., based on user interaction). When multiple cards are rendered sequentially in the simulated environment, the computing system can determine updates to the state of the stateful design elements after the first rendering. Thereafter, (e.g., when the next card in the sequence is rendered), the rendering of the stateful design elements can begin to take into account changes in the state of the stateful design elements resulting from the previous rendering in the sequence.

[0006] In embodiments, a stateful design element may correspond to a video element. When multiple cards containing a video element are rendered in a simulated environment, the computing system can determine, based on semantic analysis, that the video element will be identical in the production environment. When a card containing a video element is first rendered in the simulated environment, state information for the video element is recorded (e.g., indicating the play time when the first card is rendered). The computing system uses the state information when rendering the video element on subsequent cards in the simulated environment so that the start time of the video element follows the end time at which rendering of the video element was stopped on the previous card in the sequence.

[0007] One or more embodiments described herein provide that the methods, techniques, and actions performed by a computing device are performed programmatically or as computer-implemented methods. Programmatically, as used herein, means through the use of code or computer-executable instructions. These instructions may be stored in one or more memory resources of the computing device. Programmatically performed steps may or may not be automatic.

[0008] One or more embodiments described herein can be implemented using program modules, engines, or components. A program module, engine, or component can include a program, a subroutine, a portion of a program, or a software or hardware component that can perform one or more specified tasks or functions. As used herein, a module or component can exist on a hardware component independent of other modules or components. Alternatively, a module or component can be a shared element or process of other modules, programs, or machines.

[0009] Some embodiments described herein may generally require the use of computing devices that include processing and memory resources. For example, one or more embodiments described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, mobile phones or smartphones, tablets, wearable electronic devices, laptop computers, printers, digital photo frames, network equipment (e.g., routers), and tablet devices. Memory, processing, and network resources may all be used in connection with establishing, using, or executing any embodiment described herein (including performing any method or implementing any system).

[0010] Furthermore, one or more embodiments described herein may be implemented through the use of instructions executable by one or more processors. These instructions may be stored on a computer-readable medium. The machines illustrated or described using the following figures provide examples of processing resources and computer-readable media capable of carrying and / or executing instructions for implementing embodiments of the present invention. In particular, many machines illustrated with embodiments of the present invention include a processor and various forms of memory for storing data and instructions. Examples of computer-readable media include permanent memory storage devices such as hard drives on personal computers or servers. Other examples of computer storage media include portable storage units such as CD or DVD units, flash memory (such as found in smartphones, multifunction devices, or tablets), and magnetic memory. Computers, terminals, and network-enabled devices (e.g., mobile devices such as mobile phones) are all examples of machines and devices that utilize a processor, memory, and instructions stored on a computer-readable medium. Furthermore, embodiments may be implemented in the form of a computer program or a computer-usable carrier medium capable of carrying such a program. [Brief explanation of the drawings]

[0011] [Figure 1A] FIG. 1 illustrates an interactive graphic design system for a user's computing device, according to one or more examples. [Figure 1B] FIG. 1 illustrates a network computing system for implementing an interactive graphic design system on a user computing device, according to one or more examples. [Figure 1C] FIG. 1 illustrates a network computing system for implementing an interactive graphic design system for multiple users in a collaborative network platform, according to one or more examples. [Figure 2] FIG. 1 illustrates a simulation engine, according to one or more embodiments. [Figure 3] FIG. 1 illustrates an exemplary method for implementing a simulation environment for a graphic design system, according to one or more embodiments. [Figure 4A] FIG. 1 illustrates a design interface in which a collection of cards is provided, according to one or more embodiments. [Figure 4B] FIG. 1 illustrates a sequence of renderings generated in a simulated environment of a collection of cards, in accordance with one or more embodiments. [Figure 4C] FIG. 1 illustrates a sequence of renderings generated in a simulated environment of a collection of cards, in accordance with one or more embodiments. [Figure 4D] FIG. 1 illustrates a sequence of renderings generated in a simulated environment of a collection of cards, in accordance with one or more embodiments. [Figure 4E] FIG. 1 illustrates a semantic memory structure determined and used in connection with cards of a collection being rendered in a simulated environment, according to one or more embodiments. [Figure 5]FIG. 1 illustrates a networked computer system in which one or more embodiments may be implemented. [Figure 6] FIG. 1 illustrates a user computing device for use with one or more examples as described. DETAILED DESCRIPTION OF THE INVENTION

[0012] System Description 1A illustrates an interactive graphic design system for a user's computing device, according to one or more examples. The interactive graphic design system ("IGDS") 100 can be implemented in any one of several different computing environments. For example, in some variations, the IGDS 100 can be implemented as a client-side application running on the user computing device 10 to provide functionality as described in the various examples. In other examples, as described below, the IGDS 100 can be implemented through the use of a web-based application 80. Additionally or alternatively, the IGDS 100 can be implemented as a distributed system, such that the processes described in the various examples run on network computers (e.g., servers) and on the user device 10.

[0013] By way of example, the IGDS 100 may be implemented on a user computing device 10 to enable a corresponding user to design various types of interfaces using graphical elements. The IGDS 100 may include processes executed as or through a Web-based application 80 installed on the computing device 10. As illustrated by various examples, the Web-based application 80 may execute scripts, code, and / or other logic (“program components”) to implement the functionality of the IGDS 100. Furthermore, in some variations, the IGDS 100 may be implemented as part of a network service, and the Web-based application 80 may communicate with one or more remote computers (e.g., servers used for the network service) to execute the processes of the IGDS 100.

[0014] In some examples, the web-based application 80 obtains some or all of the program resources for implementing the IGDS 100 from a network site. Additionally or alternatively, the web-based application 80 can obtain some or all of the program resources from a local source (e.g., local memory resident on the computing device 10). The web-based application 80 can also access various types of datasets in providing the IGDS 100. The datasets can correspond to files and libraries, which can be stored remotely (e.g., on a server, associated with an account) or locally.

[0015] In an example, the web-based application 80 may support commercially available browsers such as GOOGLE CHROME (developed by GOOGLE, INC.), SAFARI (developed by APPLE, INC.), and INTERNET EXPLORER (developed by MICROSOFT CORPORATION). In such an example, the IGDS 100 processes may be implemented as scripts and / or other embedded code that the web-based application 80 downloads from a network site. For example, the web-based application 80 may execute code embedded within a web page to implement the IGDS 100 processes. The web-based application 80 may also execute scripts to retrieve other scripts and program resources (e.g., libraries) from the network site and / or other local or remote locations. As an example, the web-based application 80 may execute JAVASCRIPT® embedded in HTML resources (e.g., web pages structured according to HTML 5.0 or other versions provided under standards published by the W3C or WHATWG consortium). In some examples, the rendering engine 120 and / or other components may utilize graphics processing unit (GPU) acceleration logic, such as that provided through WebGL (Web Graphics Library) programs that execute Graphics Library Shader Language (GLSL) programs that run on the GPU.

[0016] According to an example, a user of computing device 10 operates web-based application 80 to access network sites to obtain and execute program resources to implement IGDS 100. In this manner, a user may initiate a session to implement IGDS 100 for purposes of creating and / or editing a design interface. In an example, IGDS 100 includes program interface 102, input interface 118, and rendering engine 120. Program interface 102 may include one or more processes executed to access and obtain program resources from local and / or remote sources.

[0017] In one implementation, the programmatic interface 102 can use programmatic resources associated with the web-based application 80 (e.g., an HTML 5.0 canvas), for example, to generate the canvas 122. Additionally or alternatively, the programmatic interface 102 can trigger or otherwise generate the canvas 122 using programmatic resources and data sets (e.g., canvas parameters) obtained from a local source (e.g., memory) or a remote source (e.g., a network service).

[0018] The programmatic interface 102 may also obtain programmatic resources including an application framework for use with the canvas 122. The application framework may include, for example, a data set that defines or configures a set of interactive graphic tools that are integrated with the canvas 122 and that configure the input interface 118, allowing a user to provide input for creating and / or editing the design interface.

[0019] According to some examples, the input interface 118 can be integrated with the canvas 122 and implemented as a functional layer that detects and interprets user input. The input interface 118 can, for example, use a reference to the canvas 122 to identify the on-screen location of a user input (e.g., a “click”). Furthermore, the input interface 118 can interpret a user's input action based on the location of the detected input (e.g., whether the location of the input indicates a tool selection, an object rendered on the canvas, or an area of ​​the canvas), the frequency of the detected input in a given period of time (e.g., a double-click), and / or the start and end positions of an input or series of inputs (e.g., the start and end positions of a click and drag), as well as various other input types (e.g., a right-click, a screen tap, etc.) that a user can specify through one or more input devices. In this way, the input interface 118 can, for example, interpret a series of inputs as a design tool selection (e.g., a shape selection based on the input location) or as input to define attributes (e.g., dimensions) of the selected shape.

[0020] Additionally, the programmatic interface 102 can be used to retrieve program resources and datasets, including files 101, from local or remote sources that comprise a user's active workspace. In an example, the files 101 can include a collection of cards, where the cards in the collection provide design elements for a user interface or presentation when rendered in a production environment. In an example, an individual card can represent, for example, an application screen or an application state. The retrieved datasets can include one or more cards that include design elements that collectively form a design interface, or a design interface in progress. Each file 101 can include one or more data structure representations 111 that collectively define the design interface. The files 101 can also include additional datasets associated with the active workspace. For example, as described in some examples, a workspace file can store an animation dataset that defines animation behavior, such as between objects or states, in a rendering of the canvas 122.

[0021] In an example, the rendering engine 120 uses the data structure representation 111 to render a corresponding design under editing (DUE 125) DUE on the canvas 122, where the DUE 125 reflects the graphic elements and their respective attributes provided with the individual pages of the file 101. A user can edit the DUE 125 using the input interface 118. Alternatively, the rendering engine 120 can generate a blank page of the canvas 122, and a user can generate the DUE 125 using the input interface 118. When rendered, the DUE 125 can include graphic elements, such as a background and / or a set of objects (e.g., shapes, text, images, program elements), as well as attributes of the individual graphic elements. Each attribute of a graphic element can include an attribute type and an attribute value. For objects, attribute types include shape, dimension (or size), layer, type, color, line weight, text size, text color, font, and / or other visual characteristics. Depending on the implementation, the attributes reflect properties of a two-dimensional or three-dimensional design. In this manner, the attribute values ​​of individual objects may define, for example, the size, color, position, layering, and visual characteristics of the content of elements rendered as part of the DUE 125.

[0022] Network Computing System Implementing IGDS 1B illustrates a network computing system for implementing an interactive graphic design system on a user computing device, according to one or more examples. A network computing system such as that illustrated in the example of FIG. 1B can be implemented using one or more servers that communicate with the user computing device over one or more networks.

[0023] 1B example, network computing system 150 performs operations to enable IGDS 100 to be implemented on user computing device 10. In a variation, network computing system 150 provides network services 152 to support use of IGDS 100 by user computing devices utilizing a browser or other web-based application. Network computing system 150 may include a site manager 158 that manages a website where a set of web resources 155 (e.g., web pages) are made available for site visitors. Web resources 155 may include instructions such as scripts or other logic (“IGDS instructions 157”) that are executable by a browser or web component of the user computing device.

[0024] In some variations, when computing device 10 accesses and downloads web resource 155, web-based application 80 executes IGDS instructions 157 to implement functionality as described in some examples of FIG. 1A. For example, IGDS instructions 157 may be executed by web-based application 80 to launch program interface 102 on user computing device 10. The launch of program interface 102 may occur simultaneously with the establishment of a web socket connection between program interface 102 and service component 160 of network computing system 150, for example.

[0025] In some examples, web resources 155 include logic that web-based application 80 executes to launch one or more processes of program interface 102, thereby causing IGDS 100 to obtain additional program resources and data sets for implementing functionality as described by the examples. Web resources 155 may, for example, embed logic (e.g., JAVASCRIPT code), including GPU acceleration logic, in an HTML page for download by a user's computing device. Program interface 102 may be triggered to obtain additional program resources and data sets, for example, from network services 152 and / or from local resources of computing device 10, to implement IGDS 100. For example, some of the components of IGDS 100 may be implemented through web pages that can be downloaded to computing device 10 after authentication is performed and / or when a user performs additional actions (e.g., downloading one or more pages of a workspace associated with an account identifier). Thus, in the illustrated example, the network computing system 150 can communicate the IGDS instructions 157 to the computing device 10 through a combination of network communications, including through an activity download of the web-based application 80, where the IGDS instructions 157 are received and executed by the web-based application 80.

[0026] Computing device 10 may use web-based application 80 to access a website of network service 152 and download web pages or web resources. Upon accessing the website, web-based application 80 may communicate an account identifier to service component 160 automatically (e.g., through stored credentials) or through manual entry. In some examples, web-based application 80 may also communicate one or more additional identifiers that correlate with the user identifier.

[0027] Additionally, in some examples, the service component 160 may retrieve profile information from a user profile store using a user or account identifier of the user identifier. Additionally or alternatively, the user's profile information may be determined and stored locally on the user's computing device 10.

[0028] The service component 160 can also retrieve from the file store 164 a file of an active workspace linked to a user account or identifier ("active workspace file 163"). The profile store can also identify a workspace identified with the account and / or user, and the file store 164 can store the datasets that make up the workspace. The datasets stored in the file store 164 can include, for example, the pages of the workspace, a dataset identifying constraints for the active set of workspace files, and one or more data structure representations 161 of the design being edited that can be rendered from each active workspace file.

[0029] Further, in the example, the service component 160 provides the web-based application 80 with a representation 159 of a workspace associated with the user, which representation identifies, for example, individual files associated with the user and / or user account. The workspace representation 159 can also identify a set of files, each file containing one or more pages, and each page containing objects that are part of a design interface.

[0030] On the user device 10, the user can view the workspace representation through a web-based application 80, and the user can select to open a file of the workspace through the web-based application 80. In an example, when the user selects to open one of the active workspace files 163, the web-based application 80 launches a canvas 122. For example, the IGDS 100 can launch an HTML 5.0 canvas as a component of the web-based application 80, and the rendering engine 120 can access one or more data structure representations 111 of the design interface being edited and render a corresponding DUE 125 on the canvas 122.

[0031] The service component 160 can also determine a user's permission setting or role associated with the account identifier based on the user credentials. The user's permission setting or role can determine, for example, the files the user can access. In some examples, the implementation of the rendering engine 120 on the computing device 10 can be configured based at least in part on the user's role or setting. For example, the user's ability to specify constraints for the DUE 125 can be determined by the user's permission setting, and the user can be authorized or revoked from creating constraints 145 for the DUE 125 based on their respective permission setting. Furthermore, in some variations, the response actions the user can take to resolve conflicts can be limited by the user's permission setting. For example, the user's ability to override constraints 145 can be based on the user's permission setting.

[0032] In examples, changes implemented by rendering engine 120 to DUE 125 may also be recorded with the respective data structure representation 111 for storage on computing device 10. Program interface 102 may repeatedly or continuously stream change data 121 to service component 160, with updates reflecting edits made to DUE 125 and data structure representation 111 and reflecting changes made by the user to DUE 125 and local data structure representation 111 of DUE 125. Service component 160 may receive change data 121 and then use this change data 121 to implement changes in network-side data structure representation 161. In this manner, network-side data structure representation 161 in active workspace file 163 may mirror (or synchronize) the local data structure representation 111 on user computing device 10. Once the rendering engine 120 implements changes to the DUE 125 on the user device 10, the changes may be recorded or otherwise implemented with the local data structure representation 111, and the programmatic interface 102 may stream the changes as change data 121 to the service component 160 to synchronize the local representation 111 of the DUE 125 with the network-side representation 161. This process may be performed repeatedly or continuously so that the local representation 111 of the DUE 125 and the network-side representation 161 remain synchronized.

[0033] Collaborative Network Platform 1C illustrates a network computing system for implementing an interactive graphic design system for multiple users in a collaborative network platform, according to one or more examples. In the example of FIG. 1C, the collaborative network platform is implemented by a network computing system 150 that communicates with multiple user computing devices 10, 12 over one or more networks (e.g., the World Wide Web) and implements an IGDS 100 on each computing device. While FIG. 1C illustrates an example in which two users utilize the collaborative network platform, in the illustrated example, the network computing system 150 enables collaboration on design interfaces among a larger group of users.

[0034] 1C, the user computing devices 10, 12 can be assumed to be operated by multiple users associated with a common account, each implementing a corresponding IGDS 100 to access the same workspace during their respective overlapping sessions. Thus, each of the user computing devices 10, 12 can simultaneously access the same set of active workspace files 163, and the respective programmatic interfaces 102 of the IGDSs 100 on each user computing device 10, 12 operate to establish corresponding communication channels (e.g., web socket connections) with the service components 160.

[0035] In an example, the service component 160 can communicate a copy of the active workspace file 163 to each user computing device 10, 12 such that the computing devices 10, 12 simultaneously render the DUEs 125 in the active workspace file 163. Additionally, each of the computing devices 10, 12 can maintain a local data structure representation 111 of the respective DUEs 125 as determined from the active workspace file 163. The service component 160 can also maintain a network-side data structure representation 161 obtained from the file of the active workspace 163 and consistent with the local data structure representation 111 of each of the computing devices 10, 12.

[0036] The network computing system 150 can continuously synchronize the active workspace files 163 on each user computing device. In particular, changes made by a user to the DUE 125 on one computing device 10, 12 can be immediately reflected in the rendered DUE 125 on the other user computing device 10, 12. As an example, a user at a computing device 10 can make changes to the respective DUE 125, and the respective rendering engines 120 can implement updates that are reflected in their local copies of the data structure representation 111. From the computing device 10, the program interface 102 of the IGDS 100 can stream change data 121 reflecting the user-entered changes to the service component 160. The service component 160 processes the change data 121 on the user computing device. The service component 160 can use the change data 121 to make corresponding changes to the network-side data structure representation 161. The service component 160 can also stream remotely generated change data 171 (which, in the provided example, corresponds to or reflects change data 121 received from the user device 10) to the computing device 12 to cause the corresponding IGDS 100 to update the DUE 125 rendered on that device. The computing device 12 can also use the remotely generated change data 171 to update its local data structure representation 111. The program interface 102 of the computing device 12 can receive the updates from the network computing system 150, and the rendering engine 120 can update the DUE 125 and its respective local copy 111 on the computing device 12.

[0037] The reverse process may also be implemented to update the data structure representation 161 of the network computing system 150 using change data 121 communicated from the second computing device 12 (e.g., corresponding to a user of the second computing device updating the DUE 125 rendered on the second computing device 12). The network computing system 150 may then stream remotely generated change data 171 (which, in the provided example, corresponds to or reflects the change data 121 received from the user device 12) to update the local data structure representation 111 of the DUE 125 on the first computing device 10. In this manner, the DUE 125 of the first computing device 10 may be updated in response to a user of the second computing device 12 providing user input to modify the DUE 125.

[0038] To facilitate synchronization of the data structure representations 111, 111 on the computing devices 10, 12, the network computing system 150 may implement stream connectors that merge data streams exchanged between the first computing device 10 and the network computing system 150 and between the second computing device 12 and the network computing system 150. In some implementations, the stream connectors may be implemented to allow each computing device 10, 12 to make changes to the network-side data representation 161 without the additional data duplication that may be required to process the streams from each device individually.

[0039] Additionally, over time, one or both of the computing devices 10, 12 may become out of sync with the server-side data representation 161. In such an event, each computing device 10, 12 may re-download the active workspace file 163 to resume maintenance of the data structure representation of the DUE 125 that is rendered and edited on that device.

[0040] Simulation Engine 1A-1C , by way of example, the IGDS 100 may implement a user-facing simulation engine 200. In some examples, the simulation engine 200 may implement alternative modes, including a design mode and a simulation mode. In the simulation mode, the simulation engine 200 generates simulated renderings of individual cards in a collection. The simulation engine 200 may render a sequence of cards to provide the user with a production environment simulation of the design interface in progress or during editing. In examples, the simulation engine 200 may be implemented as part of the rendering engine 120. In variations, the simulation engine 200 may be implemented through a separate component.

[0041] As described in the examples, the simulation engine 200 can implement a process for efficiently generating simulation renderings in which stateful design elements are interactive and / or dynamic, such that the stateful design elements change state in response to user input or other events as the simulation rendering is generated. Among other benefits, in the examples, such simulation renderings enable the rendering of stateful design elements to be interactive and / or dynamic, accurately replicating a production environment for the simulated design. When a stateful design element is rendered in a simulated rendering of a card, the state of the design element can change (e.g., in response to user input). For example, a stateful design element may correspond to a video element, which undergoes state changes (e.g., play time) during playback. In the examples, the simulation engine 200 renders multiple cards in which the state of the rendered stateful element transitions from card to card to more accurately simulate how the stateful element would be rendered in a production environment.

[0042] 2 illustrates an example of a simulation engine according to one or more embodiments. The simulation engine 200 is implemented in, or otherwise provided with, the IGDS 100, and enables a user to simulate how a sequence of cards would be rendered in a production environment (a "production environment rendering" or "simulation rendering"), with each card including a top-level frame containing a set of design elements. Thus, the simulation engine 200 can generate a production environment rendering as output, often utilizing multiple cards 200 in a collection 201, where the design elements of each card 202 are combined to simulate a set of production elements for a user interface or presentation in a production environment.

[0043] In some examples, the simulation engine 200 can be implemented as part of the rendering engine 120 of the IGDS 100. For example, the IGDS 100 can implement alternative modes, including a design mode and a simulation mode, in which the rendering engine 120 executes the processes of the simulation engine 200 to render as output a production environment rendering 205 that simulates the design interface during production. The production environment rendering 205 can be provided to the user devices 10, 12 to allow designs and users of the IGDS 100 to see how their designs in progress will appear in a production environment. In variations, the simulation engine 200 can be implemented as a separate component or application.

[0044] A logical hierarchical representation of cards As described in the examples, the IGDS 100 allows a user to interact with the canvas to create design elements, which can have spatial and logical relationships to one another. Design elements can be linked, for example, to have a parent / child relationship (or are also referred to as nested design elements). In the examples, nested design elements can have spatial and logical relationships to one another. For example, one design element may be nested within another, meaning that the boundary or frame of one design element (e.g., the child element) is contained within the boundary or frame of the other design element (e.g., the parent element). Furthermore, nested design elements can be logically linked, for example, such that design input to one design element triggers rules or other logic that affect the other design element. Rules or logic that affect nested design elements can serve to maintain the spatial relationships of the design elements, for example, so that one node remains a parent of another node (or one node remains a child of another node) despite resizing or repositioning input that affects the parent-child spatial relationship. Thus, for example, nested design elements may be subject to a common set of constraints as well as other functional features (e.g., auto-layout). Additionally, as another example, design input that moves one design element of a nested pair may result in the other design element being moved or resized.

[0045] Additionally, the IGDS 100 allows a user to specify a flow that specifies a sequence (including alternate sequences) between multiple cards. For example, a user can specify logical connections between a collection 201 of cards 202, where the logical connections specify a sequence. Individual cards 202 can specify alternate states of the same screen or interface, for example, and the use of such logical connections can specify state changes or flow of a user interface or presentation during production, where the state changes or flow respond to events (e.g., user input) that may occur in such a production environment. The IGDS 100 can determine and utilize a common hierarchical logical data structure ("design-mode node representation 209") to represent a collection of cards. For example, a hierarchical node representation can be maintained for the collection 201 of cards, where the representation includes a parent node and subnodes that further contain hierarchically arranged nodes. Thus, in an example, each card 202 in the collection can be represented by a root node (level 0, or top-level node), and each design element can be represented as a subnode of the root node. Within each root node, subnodes with different levels can be arranged. The top-level subnodes (i.e., level 1 nodes) of the root node may contain design elements of cards 202 that are not children of any other design elements, except for the top frame represented by the root node. Similarly, any child design elements to one of the design elements represented by the top subnodes (level 1) may be represented by a second-level subnode (i.e., level 2 node), and so on. A design mode node representation 209 may be determined for each card 202 and further combined for all cards in a collection 201. The design mode node representation 209 of a collection 201 may be provided by the IGDS 100, for example, as part of a separate panel within a tools panel of the IGDS 100.

[0046] Simulation Rendering Logic In the example, the simulation engine 200 includes processes represented by semantic decision logic 210 and simulation rendering logic 220. The simulation rendering logic 220 generates a production environment rendering 205 from each card 202 processed by the simulation engine 200, where the production environment rendering 205 includes production elements of a simulated user interface or presentation. Furthermore, the production environment rendering 205 can respond interactively or dynamically to events, such as responding to user input simulating end-user input in a production environment. Upon generating the initial production environment rendering 205 of the first selected card, the simulation rendering logic 220 generates the next production rendering from the next selected card 202 from the collection 201, and so on, such that a sequence of cards 202 is selected and used to generate each production rendering 205 of the collection. The selection of an individual card 202 for rendering can be based, for example, on user input or interaction with one of the production environment renderings 205, predefined logical connections between the cards 202, and / or other events. In this manner, a sequence of cards 202 can be dynamically selected and used to generate a production environment rendering 205. In other implementations, a sequence of cards 202 is pre-selected for rendering by simulation rendering logic 220.

[0047] Semantic Decision Logic Each card 202 can include multiple design elements, but the design elements provided on different cards 202 in collection 201 can represent the same production element. For example, multiple cards 202 in collection 201 can represent the same production environment element in different states. Simulation engine 200 implements semantic decision logic 210, which processes each card 202 rendered through execution of simulation rendering logic 220, to determine a semantic structure representative of the production element in the simulated output. In some examples, semantic decision logic 210 maintains and updates a semantic memory component 222, the structure of which corresponds to the semantic structure determined through processing of individual cards 202. The semantic structure includes node representations of the design elements of one or more cards 202 rendered by simulation rendering logic 220, and each node in the semantic structure represents a production element of the simulated user interface or presentation. In some examples, the semantic decision logic 210 processes a sequence of cards 202 to build the semantic structure of a given production rendering, and the sequence can be determined by user input, randomly, or in response to other input.

[0048] Starting with the first card in the sequence, the semantic determination logic 210 scans (i) each design element of the first card and / or (ii) the portion of the design mode node representation 209 of the collection 201 that corresponds to the particular card 202, thereby determining the semantic structure of the card's design elements. The semantic structure identifies spatial and / or logical relationships between the individual design elements of the card, such as whether design elements are nested. Additionally or alternatively, the semantic structure can identify other types of logical connections between design elements that appear on the same card. Furthermore, the semantic structure can also identify design elements that overlap each other (with or without an associated logical relationship).

[0049] In examples, the semantic structure may be in the form of a graph, with each node in the graph representing a design element of the first card 202. In some examples, the semantic determination logic 210 scans the design elements of the first card 202 to determine one or more higher-level design elements that correspond to design elements that have no parent other than the card's own upper frame. For example, a name or identifier for each higher-level design element is identified. The name / identifier (or other property or characteristic) of the higher-level design element is used to identify a root node for the card in the design mode node representation 209 of the collection 201. Once the root node is identified, a semantic data structure is determined based on the hierarchical structure of the corresponding portion of the design mode node representation 209. For example, the semantic data structure may be implemented in the form of a graph, with (i) the leftmost graph node representing the root node (i.e., level 0 of the hierarchical structure representing a card); (ii) the first level of graph nodes representing the top subnodes of the design mode node representation 209 (level 1 of the hierarchical structure corresponding to design elements that have no parent other than the card's frame); (iii) the second level of graph nodes representing the first nesting level of design elements in the hierarchical node representation 209 (level 2 of the hierarchical structure); and so on. In an example, each graph node may also be associated with the name, properties, and / or characteristics of the corresponding design element and / or node in the hierarchical logical node representation 209. In this manner, each node in the graph node structure corresponds to one of the design elements, and the connections in the graph node structure define the nested relationships between the identified design elements. In an example, the semantic determination logic 210 makes an initial determination of the semantic structure of the first structure and then generates the semantic memory component 222 accordingly.

[0050] For second and subsequent rendered cards 202, the semantic determination logic 210 scans the design elements of the second (or subsequent) card 202 to determine one or more ancestor design elements that correspond to design elements that have no parent other than the ancestor frame of the second card. A name or other identifier for each ancestor design element of the second card is identified. The name (or other property or characteristic) of the ancestor design element can be used to identify a root node for the second card (and subsequent cards) in the design mode node representation 209. Once the root node for the second card is identified, a semantic data structure for the respective card can be determined based at least in part on the hierarchical structure of the corresponding portion of the design mode node representation 209. The semantic structure of the second or subsequent card can be compared to an existing semantic structure, as provided by the semantic memory component 222, to determine design elements of the second or subsequent card that are not represented in the existing semantic structure. For each identified design element not represented in the existing semantic structure, semantic decision logic 210 is executed to construct the identified design element as an additional node in the semantic structure and update semantic memory component 222 accordingly. This process is repeated for each card in collection 201 rendered by simulation engine 200, such that the semantic structure is updated to include the newly identified node in its graph-form structure. In this way, semantic memory component 222 may be in graph form, with nodes representing production elements of the simulated design.

[0051] In an example, the semantic decision logic 210 can identify stateful design elements that may be present within an individual card 202. A stateful element refers to a design element that can undergo state changes in a production (or simulated production) environment. A stateful element may correspond to any design element that can change due to a change in properties when rendered in a simulation. As an example, a stateful element may correspond to a video element, and the state of the video element may include playback time. In other variations, the stateful properties of a media file may correspond to playback speed or format, volume, or any other detectable characteristic of the media output that can change as a result of user input or other events. For example, in connection with rendering a given card 202, the simulation rendering logic 220 may render the video element for a period of time that may change based on user input. Upon receiving user input, playback of the video element is stopped and the next card is rendered. In such a case, the playback time of the video element at a point in time before receiving the user input may define the state of the video element immediately before the next card is rendered. The simulation rendering logic 220 may record the playback time as state information via the semantic memory component 222. When the next card is loaded, the simulation rendering logic 220 can identify a stateful design element (e.g., a video element) that is identical to a design element of the preceding card using the semantic structure recorded by the semantic memory component 222. Based on this determination, the simulation rendering logic 220 can use the state information recorded by the simulation rendering logic 220 when rendering, or beginning to render, the identified stateful design element. In the example of a video element, this would result in the video element starting to play from its recorded play time.

[0052] Although some of the described examples are specific to video or media, the described embodiments can be applied to other types of stateful design elements. For example, an input selector (e.g., a date field selector) can allow a user to select a date from a set of possible options (e.g., by month). In a particular state, a user's selection of a particular option (e.g., a particular date) can be highlighted. In an example, the state change of an input selector can be simulated through the rendering of multiple cards, for example, to simulate a user scrolling through multiple options (e.g., multiple dates on a calendar view).

[0053] As another example, a stateful design element may correspond to a checkbox that can change between a checked and unchecked state. If the checkbox is reused in a subsequent frame, the state of the checkbox may be preserved and displayed in simulated renderings of subsequent cards. As another example, a design element may render a media field with a changing viewing angle. For example, the media feed may correspond to live game content that may be rendered, allowing the user to view the rendering from various angles. Alternatively, the media feed may be generated by a camera that can be rotated to change the viewing angle. In such an example, the simulated rendering between cards may identify the feed as a stateful element, changing state to change the viewpoint or viewing angle.

[0054] methodology Figure 3 illustrates an exemplary method for implementing a simulation environment for a graphic design system, according to one or more embodiments. The method as described in the example of Figure 3 may be implemented using a simulation engine for a graphic design system as described in the examples of Figures 1A-1C and 2. Accordingly, in describing the example of Figure 3, reference may be made to elements of Figures 1A-1C and 2 to describe functionality for implementing the steps or sub-steps being described.

[0055] Referring to FIG. 3 , in step 302, the simulation engine 200 selects an initial design interface card from a collection of cards for rendering. Depending on the implementation, the selection of the initial card can be based on user selection, a pre-determination, or a response to some other event. As described in various examples, one or more users can utilize the IGDS 100 to generate a collection of cards, each card including design elements that depict or otherwise represent a particular operational state of a user interface or presentation during production. For example, cards in the collection can depict screens of an application (e.g., a mobile app) beginning with a home screen. In such an example, each card can represent (i) a particular application screen (e.g., a home screen) or (ii) an application screen in a particular state (e.g., a start screen with interactive features that receive user input). Thus, in such an example, multiple cards can represent a single application screen, and design elements on different cards can represent the same element of the user interface or presentation (“production element”). Additionally, designers can use connectors to specify flow between cards, where the flow specifies a sequence between subsets of cards based on user interaction or other events. The simulation engine 200 can, for example, interactively render individual cards based on the connectors and / or design elements specified on the individual cards.

[0056] In step 310, simulation engine 200 processes information associated with the selected card and / or initial card 202 or collection 201 to determine a semantic structure for the card. The semantic structure may identify the design elements of the initial card and one or more spatial and / or logical relationships between the design elements. In some examples, the semantic structure identifies nested relationships between the design elements of the initial card. The semantic structure may be in the form of a graph, with each design element of the initial card represented by a node in the graph and nested relationships between the design elements represented by connectors in the graph. In this manner, the semantic structure may represent semantic identifiers for individual nodes of the semantic structure (representing the design elements of the initial card). The semantic identifier for each node may correspond, for example, to the name of the corresponding design element combined with the node's node path within the semantic structure (e.g., relative to a root node representing the card's container). Simulation engine 200 may also associate additional properties with each node, such as attributes identifying the type, color, shading, line weight, or other properties of the corresponding design element. The information used to determine the semantic structure can include a hierarchical design mode representation 209 of the collection 201, where the hierarchical arrangement specifies nested relationships between the design elements.

[0057] In step 312, simulation engine 200 may also identify one or more design elements that represent corresponding stateful design elements of the initial card. As described in other examples, a stateful design element refers to a design element that can undergo state changes in the simulation environment. As an example, a stateful design element may correspond to a playable video element, where the play time of the video element defines the state of the video element.

[0058] Further, in step 320, the simulation engine 200 can generate a rendering of the production environment using the initial card. The generation of the rendering can be performed in parallel or simultaneously as the semantic structure of the initial card is determined. Based on the implementation, the production environment rendering can receive user interactions to simulate actual user interactions in the production environment. In this manner, the state of a stateful design element can change as the initial card is rendered. For example, the stateful design element can correspond to a video element that is playable when the production environment rendering of the initial card begins. The user can interact with the video element when the production environment rendering is generated, for example, to start playback, pause, stop playback, or perform other interactions.

[0059] In step 322, simulation engine 200 records the state of each of one or more stateful design elements after the rendering of the initial card is generated. After the rendering is generated, a user can interact with the rendering to change the state of the stateful elements. For example, the stateful elements can correspond to video elements, and simulation engine 200 can play a video in conjunction with the rendering of the initial card. The state information can be recorded in association with a semantic identifier of the stateful design element. For example, the semantic structure can be in the form of a graph, and the stateful design element can be identified by the node path of the stateful element within the graph representation of the semantic structure. The node path can identify, for example, the name (or other identifier) ​​of each upstream node corresponding to each parent design element of the stateful design element, up to the root node. In this way, the semantic identifier of the stateful element can identify the node location or path of the stateful element as well as the name of the corresponding design element.

[0060] In an example, simulation engine 200 maintains a semantic memory component 222 that records the determined semantic structure. In particular, the recorded state information can include semantic identifiers of individual design elements, including stateful design elements. For stateful design elements, simulation engine 200 can record the determined state information in association with the semantic identifier of a corresponding node in the semantic structure.

[0061] In step 330, the simulation engine 200 identifies the next card for simulation rendering based on, for example, user input or detected events and / or a predefined workflow, which may be specified, for example, by connectors that a user may specify between cards 202 in the collection.

[0062] In step 332, the simulation engine 200 processes information associated with the next card and / or the initial card 202 or collection 201 to determine a semantic structure for the next card. Similar to the initial card, the semantic structure may identify the design elements of the initial card as well as one or more spatial and / or logical relationships (e.g., nesting relationships) between the design elements. The semantic structure may also be constructed based at least in part on the hierarchical design mode representation 209 of the collection 201. Additionally, semantic identifiers for individual nodes of the next card may be determined from the semantic structure based on the name of the corresponding design element, the node path of the node within the semantic structure, and / or other properties (e.g., the type, color, shading, line weight, or other properties of the corresponding design element).

[0063] In step 334, simulation engine 200 may update the existing determined semantic structure based on the semantic structure determined for the next card. In determining the semantic structure, simulation engine 200 may determine whether individual design elements of the next card are represented by an existing semantic structure determined from the first card or any other preceding card. The existing semantic structure may be recorded in semantic memory component 222. In an example, simulation engine 200 compares the semantic identifiers of the individual design elements of the next card with the semantic identifiers of the existing semantic structure (recorded in semantic memory component 222). For each design element of the next card, if a corresponding semantic identifier exists in the existing semantic structure, simulation engine 200 does not update the structure of semantic memory component 222. If a corresponding semantic identifier does not exist in the existing semantic structure, simulation engine 200 updates the semantic memory structure to include an additional node representing the identified design element.

[0064] In step 336, the simulation engine 200 generates a production environment rendering of the next card. The generation of the rendering can be performed in parallel or simultaneously with the determination of the semantic structure. In generating the rendering, the simulation engine accesses the semantic memory component 222 to determine information such as the state of individual design elements that are stateful. For stateful design elements, in step 338, the simulation engine 200 identifies the state information associated with the corresponding semantic identifier and renders the design element to reflect the state represented by the state information. Thus, for example, if the first card includes a video element that plays for a given time interval, the rendering of the next card will begin playing the video element at the end of the given time interval.

[0065] Additionally, in step 340, the simulation engine 200 may update state information for the stateful design element while the rendering of the next card is in progress. The update of the state information for the design element may be based, for example, on user input and / or other events that occur after the rendering is generated. Thus, for example, if a video element continues playing for a second time interval, the simulation engine 200 updates the semantic memory component 222 so that the semantic identifier associated with the video element reflects the current playback state of the video element. In step 342, it is determined whether another card in the collection is to be rendered in the simulation. This determination may be based, for example, on logical relationships between cards, user interaction with the rendered card during the simulation, and / or the structure, placement, or number of cards. If it is determined that there is another card to render in the simulation (“No”), steps 330-340 are repeated for the next card.

[0066] In examples, steps 330-340 may be repeated for each additional card rendered by simulation engine 300 until rendering of collection 201 in the simulation environment is finished (step 342). If step 342 determines that no more cards will be simulated (or the simulation is finished), the process may end. In some examples, semantic memory component 222 may be reset for the next simulation, which may occur for the same collection or a different collection.

[0067] Design Interface and Simulation Examples Figure 4A illustrates a design interface in which a collection of cards is presented, according to one or more embodiments. Figures 4B-4D illustrate a sequence of renderings of a collection of cards generated in a simulated environment. Figure 4E illustrates a semantic memory structure determined and used in connection with cards of a collection being rendered in a simulated environment, according to one or more embodiments. In describing the examples of Figures 4A, 4B-4D, and 4E, reference may be made to elements of Figures 1A-1C and 2 for purposes of illustrating functionality for implementing the described examples.

[0068] Referring to FIG. 4A, design interface 400 includes a first card 402, a second card 412, and a third card 432, which collectively form a card collection 401. Connections 405, 415 can be combined to specify the order in which individual cards are rendered in a production environment. This order can be determined by events or user input. IGDS 100 can maintain design mode node representations 209 of the design elements of collection 401. Each design element can be identified as a node on the design mode node representations 209, with each card 402, 412, 432 having a corresponding root node from which other nodes representing that card's design elements are arranged hierarchically to reflect the nested relationships.

[0069] In the illustrated example, a first card 402 includes a first video element 410 nested within several other design elements. The other design elements include a design element 404 representing a video player, which is also nested within a design element 406 representing page content, both of which are nested within the frame of the first card 402. The first card 402 also includes a second video design element 420, which is nested within another design element 418 representing a second video player, which is also nested within another design element 416 representing a suggestion box. The suggestion box design element 416 is shown nested within the page content design element 406. The first card 402 also includes a third video design element 430, which is nested within another design element 428 representing a third video player, which is in turn nested within another design element 426 representing a second suggestion box 426, which is in turn nested within the page content design element 406.

[0070] 4A, the second card 412 includes a text element 431. The third card 432 includes a video element 438, which is nested within a design element 436 representing a video player, which is also nested within a design element 434 representing the page content.

[0071] 4B-4D illustrate how simulation engine 200 sequentially generates renderings of cards 402, 412, and 432. Referring to FIG. 4B, simulation engine 200 generates a production environment rendering 480 of a first card 402. In the illustrated example, rendering 480 of first card 402 results in video playback of video element 410 for a given interval (e.g., 1 second), after which simulation engine 200 generates rendering 482 of the next card 412, as shown in FIG. 4C. In the illustrated example, simulation engine 200 generates a production environment rendering 484 based on a third card 432. Rendering 484 of third card 432 may include rendering of video element 438.

[0072] According to an example, simulation engine 200 generates and maintains a semantic structure for collection of cards 401. Maintaining the semantic structure allows simulation engine 200 to determine which design elements of cards 402, 412, and 432 are semantically identical in a production environment. For stateful design elements, simulation engine 200 can record state information (e.g., reflecting playtime) in association with semantic identifiers of node elements of the semantic structure that represent the design elements of the cards in the collection.

[0073] Figure 4E illustrates a semantic structure determined in accordance with one or more embodiments for the example of Figures 4A, 4B-4D. Referring to Figure 4E, simulation engine 200 determines the semantic structure in connection with simulation engine 200 generating a production environment rendering 480 of a first card. In the illustrated example, the semantic structure includes (in graph form): (i) a root node 440 (representing a card-level design element); (ii) a page content node 442 connected as a child to the root node 440, which represents the page content design element 406; (iii) a video player node 444 connected as a child node to the page content node 442, which represents the video player design element 404; (iv) a first suggestion box node 454 connected as a child node to the page content node 442, which represents the suggestion box design element 416; and (v) a second suggestion box node 454 connected as a child node to the page content node 442, which represents the second suggestion box design element 426. (vi) a video element node 446 connected as a child node to the first video player node 444 and representing the first video design element 410; (vii) a second video player node 456 connected as a child node to the first suggestion box node 454 and representing the second video player 418; (viii) a third video player node 466 connected as a child node to the second suggestion box node 464 and representing the third video player 428; (ix) a second video element node 458 connected as a child node to the second video player node 456 and representing the second video design element 420; and (x) a third video element node 468 connected as a child node to the third video player node 466 and representing the third video design element 430.

[0074] Simulation engine 200 processes third card 432 by determining the semantic structure of third card 432. Based on the semantic structure determination of third card 432, it is determined that video element 438 of third card 432 has the same semantic identifier as the video element of first card 402. Based at least in part on design mode node representation 209, the semantic identifier of each of video element 410 and video element 438 is determined to be: "Page Content" node-->"Video Player" node-->"Video Element" 410 may include:

[0075] Based on the semantic identifier, the simulation engine 200 can determine that the video elements 410, 438 are the same and can use the state information recorded with the video element 410 after the initial rendering to begin rendering the video element 438 when the rendering of the third card is generated.

[0076] Referring to the illustrated example, the rendering of video element 410 on first card 402 will be the same as the rendering of video element 438 on third card 432. However, in design mode, video element 410 and video element 438 are different design elements, e.g., have different node identifiers than design mode node representation 209. In a conventional approach, video elements 410 and 438 may be rendered to appear the same in a production environment, but the video elements may not be recognized as the same. Thus, in a conventional approach, when video element 438 is rendered in a simulation environment, video element 438 may start from the beginning rather than from the point where playback stopped after rendering 480 was generated.

[0077] In contrast, in the described embodiment, simulation engine 200 can recognize that video elements 410 and 438 are semantically the same, meaning that in a production environment, video element 410 in first rendering 480 is the same as video element 438 in second / third rendering 484. Thus, as described, simulation engine 200 records state information reflecting the playback state of video element 410, for example, when rendering 480 is stopped or finished (e.g., when rendering 482 is generated). When rendering 484 is generated, simulation engine 200 uses the recorded state information to begin playing video element 438 from the playback time where playback stopped or finished in first rendering 480. Among other technical advantages, rendering of card sequences results in a more accurate simulation of the production rendering.

[0078] Network Computer System FIG. 5 illustrates a computer system on which one or more embodiments may be implemented. Computer system 500 may be implemented, for example, on a server or combination of servers. For example, computer system 500 may be implemented as network computing system 150 of FIGS. 1A-1C. Additionally, in some examples, computer system 500 may provide instructions to user devices to enable the user devices to implement the functionality of IGDS 100. Additionally, computer system 500 may provide instructions to user devices or otherwise perform operations to implement an exemplary method (or steps therein) such as that described in FIG. 3.

[0079] In one implementation, computer system 500 includes processing resources 510, memory resources 520 (e.g., read-only memory (ROM) or random access memory (RAM)), one or more instruction memory resources 540, and a communication interface 550. Computer system 500 includes at least one processor 510 for processing information stored with memory resources 520, such as provided by random access memory (RAM) or other dynamic storage device, to store information and instructions executable by processor 510. Memory resources 520 may also be used to store temporary variables or other intermediate information during execution of instructions executed by processor 510.

[0080] The communication interface 550 allows the computer system 500 to communicate with one or more user computing devices through one or more networks (e.g., cellular networks) through the use of a network link 580 (wireless or wired). Using the network link 580, the computer system 500 can communicate with one or more computing devices, special purpose devices and modules, and / or one or more servers.

[0081] In an example, the processor 510 may execute service instructions 522 stored with the memory resources 520 to enable the network computing system to implement the network service 152 and operate as the network computing system 150 in an example such as that described with reference to Figures 1A-1C.

[0082] Computer system 500 may also include additional memory resources ("instruction memory 540") for storing executable instruction sets ("IGDS instructions 545") embedded in web pages and other web resources to enable user computing devices to implement functionality as described in IGDS100.

[0083] As such, the examples described herein relate to the use of computer system 500 for performing the techniques described herein. According to one aspect, the techniques are performed by computer system 500 in response to processor 510 executing one or more sequences of one or more instructions contained in memory 520. Such instructions may be read into memory 520 from another machine-readable medium. Execution of the sequences of instructions contained in memory 520 causes processor 510 to perform the process steps described herein. In alternative implementations, hardwired circuitry may be used in place of or in combination with software instructions to implement the examples described herein. Thus, the described examples are not limited to any specific combination of hardware circuitry and software.

[0084] User Computing Device 6 illustrates a user computing device for use in one or more examples, as described. In examples, user computing device 600 may correspond to, for example, a workstation, desktop computer, laptop, or other computer system having graphics processing capabilities suitable for rendering design interfaces and enabling graphic design work. In variations, user computing device 600 may correspond to a mobile computing device, such as a smartphone, tablet computer, laptop computer, VR or AR headset device, etc.

[0085] In the example, computing device 600 includes a central or main processor 610, a graphics processing unit 612, memory resources 620, and one or more communication ports 630. Computing device 600 can use main processor 610 and memory resources 620 to store and launch a browser 625 or other web-based applications. A user can operate browser 625 to access a network site of network service 152 using communication port 630, where the user can download one or more web pages or other resources 605 (see FIGS. 1A-1C ) for network service 152. Web resources 605 can be stored in active memory 624 (cache).

[0086] As described in various examples, the processor 610 can detect and execute scripts and other logic embedded in web resources to implement the IGDS 100 (see FIGS. 1A-1C). Additionally, the processor 610 can execute scripts or instructions to perform exemplary methods, such as those described in the example of FIG. 3. In some examples, some of the scripts 615 embedded in the web resources 605 can include GPU-accelerated logic executed directly by the GPU 612. The main processor 610 and the GPU can combine to render an editing design interface (“DUE 611”) on the display component 640. The rendered design interface can include web content from the browser 625 and design interface content and functional elements generated by the scripts and other logic embedded in the web resources 605. By including scripts 615 that can be executed directly on the GPU 612, the logic embedded with the web resources 615 can better execute the IGDS 100, as described in various examples.

[0087] conclusion Although embodiments are described in detail herein with reference to the accompanying drawings, it should be understood that the concepts are not limited to those precise embodiments. Accordingly, it is intended that the scope of the concepts be defined by the following claims and their equivalents. Furthermore, it is contemplated that particular features described individually or as part of an embodiment can be combined with other individually described features or with parts of other embodiments, even if the other features and embodiments do not refer to the particular feature. Thus, the absence of a description of a combination should not exclude entitlement to such a combination.

Claims

1. one or more processors; memory for storing instructions and A computer system comprising: The one or more processors execute the instructions to: Implementing an interactive graphic design system to allow a user to create multiple cards for a user interface, the interactive graphic design system providing a simulation mode in which the multiple cards can be rendered individually to simulate the user interface in a production environment; and providing the simulation mode by generating a simulated rendering of a card sequence of the plurality of cards, the card sequence including a first card and a second card; Perform an operation that includes The generating of the simulated rendering of the user interface includes: (i) processing each card in the card sequence to determine a semantic structure of the card sequence, the semantic structure including a plurality of nodes, each node of the plurality of nodes representing a production element of the simulated user interface during production; (ii) determining, based on the determined semantic structure, whether a stateful design element of each of the first card and the second card in the card sequence represents the same production element of the user interface; (iii) recording the state of the stateful design element after the first card in the card sequence is rendered; and (iv) commencing rendering of the second card in the card sequence with the stateful design element having a state based on the recorded state.

2. 2. The computer system of claim 1, wherein the stateful design elements include a video element that changes the state of the simulation mode by playing a video.

3. A computer system as described in claim 2, wherein the step of recording the state of the stateful design element includes a step of determining the state for the stateful design element immediately before rendering the second card in the card sequence.

4. further comprising a semantic memory component; the one or more processors record the determined state of the stateful design element in the semantic memory component.

4. The computer system of claim 3.

5. The operation is 5. The computer system of claim 1, further comprising: rendering the second card in the sequence of cards using the stateful design element initially having a state based on the determined state information; and updating the state of the stateful design element after the second card has been rendered.

6. A computer system as described in any one of claims 1 to 4, wherein when the first card during the simulation rendering is rendered, the state of the stateful design element changes in response to user interaction received for the stateful design element.

7. The computer system of claim 6, wherein the stateful design element includes a video element and the user interaction includes one of playing and stopping playback by the user after the first card is rendered.

8. The computer system of claim 2, wherein the recorded state includes a playback time when rendering of the first card stopped, and the start of rendering of the second card includes the start of playback of the video element at a start time based on the playback time when rendering of the first card stopped.

9. The operation is determining whether the stateful design elements of each of the first card and the second card in the card sequence represent the same production element based on the determined semantic structure includes determining semantic identifiers of the stateful design elements of the first card and the second card, and determining whether the semantic identifiers are the same; The computer system according to any one of claims 1 to 4.

10. 1. A computer-implemented method comprising: implementing an interactive graphic design system to allow a user to create multiple cards for a user interface, the interactive graphic design system providing a simulation mode in which the multiple cards can be rendered individually to simulate the user interface in a production environment; providing the simulation mode by generating a simulated rendering of a card sequence of the plurality of cards, the card sequence including a first card and a second card; Including, The generating of the simulated rendering of the user interface includes: (i) processing each card in the card sequence to determine a semantic structure of the card sequence, the semantic structure including a plurality of nodes, each node of the plurality of nodes representing a production element of the simulated user interface during production; (ii) determining, based on the determined semantic structure, whether a stateful design element of each of the first card and the second card in the card sequence represents the same production element of the user interface; (iii) recording the state of the stateful design element after the first card in the card sequence is rendered; and (iv) commencing rendering of the second card in the card sequence with the stateful design element having a state based on the recorded state.

11. The method of claim 10 , wherein the stateful design elements include a video element that changes the state of the simulation mode by playing a video.

12. The method of claim 11, wherein the step of recording the state of the stateful design element includes a step of determining the state for the stateful design element immediately before rendering the second card in the card sequence.

13. 13. The method of any one of claims 10 to 12, further comprising: rendering the second card in the sequence of cards using the stateful design element initially having a state based on the determined state information; and updating the state of the stateful design element after the second card has been rendered.

14. The method described in claim 13, wherein when the first card in the simulation rendering is rendered, the state of the stateful design element changes in response to a user interaction received for the stateful design element.

15. The method of claim 14, wherein the stateful design element includes a video element and the user interaction includes one of playing and stopping playback by the user after the first card is rendered.

16. The method of claim 15, wherein the recorded state includes a playback time when rendering of the first card stopped, and wherein starting rendering of the second card includes starting playback of the video element at a start time based on the playback time when rendering of the first card stopped.

17. A method according to any one of claims 10 to 12, wherein the state of the stateful design element is recorded in a semantic memory component.

18. A method according to any one of claims 10 to 12, wherein determining whether the stateful design elements of each of the first card and the second card in the card sequence represent the same production element based on the determined semantic structure includes determining semantic identifiers of the stateful design elements of the first card and the second card, and determining whether the semantic identifiers are the same.

19. A method according to any one of claims 10 to 12, wherein the method is executed by a networked computer system in communication with one or more user computing devices to implement the interactive graphic design system.

20. A method according to any one of claims 10 to 12, wherein the method is executed by a user computing device that implements the interactive graphic design system.

21. The method of claim 10, wherein the method is performed by instructions stored on a non-transitory computer-readable medium.

22. A non-transitory computer-readable medium storing instructions, comprising: When executed by one or more processors of a computing system, the computing system is implementing an interactive graphic design system to allow a user to create multiple cards for a user interface, the interactive graphic design system providing a simulation mode in which the multiple cards can be rendered individually to simulate the user interface in a production environment; providing the simulation mode by generating a simulated rendering of a card sequence of the plurality of cards, the card sequence including a first card and a second card; Perform an operation that includes a step of generating the simulated rendering of the user interface, the step of: (i) processing each card in the card sequence to determine a semantic structure of the card sequence, the semantic structure including a plurality of nodes, each node of the plurality of nodes representing a production element of the simulated user interface during production; (ii) determining, based on the determined semantic structure, whether a stateful design element of each of the first card and the second card in the card sequence represents the same production element of the user interface; (iii) recording a state of the stateful design element after the first card in the card sequence is rendered; and (iv) commencing rendering of the second card in the card sequence with the stateful design element having a state based on the recorded state.

23. 23. The non-transitory computer-readable medium of claim 22, wherein the stateful design elements include a video element that changes the state of the simulation mode by playing a video.

24. The step of recording the state of the stateful design element, comprising: rendering the first card in the sequence of cards in the simulated environment; determining, in the simulation environment, state information for the stateful design element immediately prior to rendering the second card in the sequence of cards; recording the determined state information in association with a semantic identifier of the stateful design element; 24. The non-transitory computer-readable medium of claim 23, comprising:

25. A non-transitory computer-readable medium as described in any one of claims 22 to 24, wherein when the first card during the simulation rendering is rendered, the state of the stateful design element changes in response to a user interaction received for the stateful design element.

Images