Design space integrated with cue space for machine learning model

Abstract

In various embodiments, a computer-implemented method for displaying a cue space, the method comprising displaying a design space comprising one or more design objects; receiving a selection of a current location within the design space; and displaying the prompt space at a placement position in the design space based on the current position. In other embodiments, a computer-implemented method for displaying a cue history, the method comprising: displaying a design space comprising a first design object; displaying a first prompt history mark in the design space, wherein the first prompt history mark represents a first prompt history associated with the first design object; and in response to receiving a selection for viewing the first prompt history, displaying the first prompt history in a prompt space within the design space.

Description

[0001] Cross-reference to related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 516,666, filed July 31, 2023, entitled “GRAPHICAL USER INTERFACE WITH COMBINED CURSOR AND PROMPT FOR ARTIFICIAL INTELLIGENCE MODELS,” and claims priority to U.S. Patent Application No. 18 / 748,933, filed June 20, 2024, entitled “DESIGN SPACE WITH INTEGRATED PROMPT SPACE FOR MACHINE LEARNING MODELS.” The subject matter of these related applications is hereby incorporated herein by reference. Technical Field

[0002] The various embodiments relate generally to computer-aided design and artificial intelligence, and more specifically to design spaces with integrated hint spaces for machine learning models. Background Technology

[0003] The design exploration of three-dimensional (3D) objects typically refers to a phase in the design process during which designers generate and modify various 3D objects within the overall 3D design. As is well known in practice, manually generating and modifying even relatively simple 3D objects can be extremely labor-intensive and time-consuming. Because the time allocated to generating 3D designs, which often consist of a large number of 3D objects, is usually limited, designers typically spend a limited amount of time generating and modifying each 3D object, which often reduces the overall quality of the final 3D design. Therefore, various traditional computer-aided design (CAD) applications have been developed to attempt to automate more comprehensively how 3D objects are generated and evaluated.

[0004] One approach to automating the generation and modification of 3D objects within a CAD application involves implementing an artificial intelligence (AI) model (such as a generative machine learning (ML) model) to provide design guidance to the user or automatically synthesize or modify 3D objects in response to user-provided prompts. Prompts provided to the AI ​​model can take the form of a query or design problem statement specifying one or more design characteristics that the generated or modified 3D object should adhere to. Prompts can include any number of quantitative objectives, physical objects, physical and functional constraints, and / or mechanical and geometric quantities to guide the AI ​​model on how to generate or modify the 3D object. The AI ​​model generates responses to the prompts, such as a natural language text response satisfying the query or design characteristics specified in the prompt (displayed in the prompt space) and / or the generated or modified 3D object (displayed in the design space).

[0005] Typically, CAD applications generate a graphical user interface (GUI) that includes a design space and a separate tooltip space for interacting with the AI ​​model. The design space displays the overall 3D design, which includes one or more 3D objects, including those automatically generated or modified by the AI ​​model. The tooltip space receives user input used to generate tooltips for the AI ​​model. The tooltip space also displays a tooltip history of all interactions with the AI ​​model, including user input used to generate various tooltips and textual responses from the AI ​​model to the tooltips. In conventional approaches, the tooltip space is often separate from and located outside the design space. For example, the tooltip space may include panels / windows within the GUI, located to the left, right, top, or bottom of the design space, while the design space itself may also include panels / windows within the GUI.

[0006] One drawback of the above method is that, because the cue space is separate from and located outside the design space, the user's focus often shifts away from the different 3D objects within the design space when interacting with it. Ideally, the 3D objects within the design space should be the core focus of the user's design work, and distraction from core design work within the design space can be counterproductive to an efficient design process. Another drawback is that, because the cue space is separate from and located outside the design space, users may have difficulty understanding the context of any given cue interaction when the cue space displays the history of cue interactions. For example, after extensive interactions with an AI model, a typical cue history often includes a long list of cue interactions, but lacks information about which specific 3D objects or locations within the design space are associated with which cue interactions are included in the cue history. Therefore, users often cannot interpret the design process or previous interactions with the AI ​​model from the cue history.

[0007] As explained above, there is a need in the field for more effective technologies for interacting with AI models when using CAD applications. Summary of the Invention

[0008] In various embodiments, a computer-implemented method for displaying a prompt space includes displaying a design space comprising one or more design objects; receiving a selection of a current position within the design space; and displaying the prompt space at a placement location within the design space based on the current position.

[0009] At least one technical advantage of the disclosed technology over existing technologies is that it integrates the cue space used for interacting with AI models into the design space of a CAD application, which reduces the amount of user experience distraction when performing design operations within the design space. In this respect, the disclosed technology enables the cue space to be displayed near the currently selected location within the design space, such as the location associated with the currently selected 3D object or an empty location. Spatially placing the cue space within the design space in relation to the user's current focus reduces distraction from performing design work within the design space and promotes a more efficient design process compared to conventional implementations where the cue space is separated from and located outside the design space. These technical advantages provide one or more technical advancements superior to existing methods.

[0010] In various embodiments, a computer-implemented method for displaying a prompt history includes: displaying a design space including a first design object; displaying a first prompt history marker within the design space, the first prompt history marker indicating a first prompt history associated with the first design object; and displaying the first prompt history in a prompt space within the design space in response to receiving a selection to view the first prompt history.

[0011] At least one technical advantage of the disclosed technology over the prior art is that it records separate and distinct cue histories for separate and distinct 3D objects and empty locations within the design space. In this regard, in response to a user request to view the cue history associated with a selected 3D object or empty location within the design space, the cue history is automatically displayed near the selected 3D object or empty location within the design space. Therefore, by using the disclosed technology, displaying the cue history associated with only the selected 3D object or empty location allows users to more easily review and understand previous design processes and interactions with the AI ​​model regarding cue history for the selected 3D object or empty location compared to conventional implementations. These technical advantages provide one or more technical advancements superior to existing methods. Attached Figure Description

[0012] By referring to various embodiments, one can understand the above-described features of the various embodiments in a more detailed description of the inventive concept briefly outlined above, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only typical embodiments of the inventive concept and should therefore not be construed as limiting the scope in any way, and other equivalent embodiments exist.

[0013] Figure 1 It is a conceptual illustration of a system configured to implement one or more aspects of various embodiments; Figure 2 According to various embodiments Figure 1 More detailed illustrations of the design exploration application; Figure 3 According to various embodiments Figure 2 A concept map from the history table; Figure 4 According to various embodiments Figure 2 An exemplary illustration of the initial selection of design objects within the design space; Figure 5 According to various embodiments Figure 4 An exemplary illustration of the interaction mode of the prompt space within the design space; Figure 6 According to various embodiments Figure 5 An exemplary illustration of an ML response shown in the prompt space; Figure 7 According to various embodiments Figure 5 An exemplary illustration of additional user input shown in the prompt space; Figure 8 According to various embodiments Figure 5 An exemplary illustration of the additional ML response shown in the prompt space; Figure 9 According to various embodiments Figure 4 An example illustration of the prompt history markers displayed within the design space; Figure 10 According to various embodiments Figure 2 An exemplary illustration of assembly-level prompt history markers displayed within the design space; Figure 11 According to various embodiments Figure 10 An exemplary illustration of a set of part-level tooltips and history markers displayed within the design space; Figure 12 According to various embodiments Figure 10 An exemplary illustration of a set of component-level tooltips and history markers displayed within the design space; Figure 13 According to various embodiments Figure 2 An exemplary illustration of the viewpoint image displayed in the prompt space; Figure 14 A flowchart illustrating the steps of a method for integrating a cue space within a design space, according to various embodiments; Figure 15 A flowchart illustrating method steps for displaying prompt history within a prompt space according to various embodiments is provided; and Figure 16 An architecture for a system in which various embodiments can be implemented is described. Detailed Implementation

[0014] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details. For purposes of explanation, multiple instances of similar objects are symbolized where necessary using reference numerals identifying the objects and bracket markers identifying instances.

[0015] System Overview Figure 1 This is a conceptual diagram of a system 100 configured to implement one or more aspects of various embodiments. As shown, in some embodiments, system 100 includes, but is not limited to, a client device 110, a server device 160, one or more remote machine learning (ML) models 190, and one or more remote context databases 194.

[0016] Client device 110 includes, but is not limited to, processor 112, one or more input / output (I / O) devices 114, and memory 116. Memory 116 includes, but is not limited to, a graphical user interface (GUI) 120, a design exploration application 130, and local data storage 140. Local data storage 140 includes, but is not limited to, one or more data files 142, one or more design objects 144, context data 146, and / or one or more snapshot images (not shown). Server device 160 includes, but is not limited to, processor 162, one or more I / O devices 164, and memory 166. Memory 166 includes, but is not limited to, an intent management application 170, one or more trained ML models 180, and a design history 182. In some other embodiments, system 100 may include any number and / or type of other client devices, server devices, remote ML models, databases, or any combination thereof.

[0017] Any number of components of System 100 can be distributed across multiple geographical locations, or in any combination within one or more cloud computing environments. For exampleThis is implemented within encapsulated shared resources, software, and data. In some embodiments, client device 110 and / or zero or more other client devices (not shown) may be implemented as one or more computing instances in a cloud computing environment, as part of any other distributed computing environment, or implemented independently. In various embodiments, client device 110 may be used with any number and / or type of other devices (… For example (One or more other computing instances and / or display devices) are integrated into the user device. Some examples of user devices include, but are not limited to, desktop computers, laptop computers, smartphones, and tablets.

[0018] Generally, client device 110 is configured to implement one or more software applications. For illustrative purposes only, each software application is described as residing in memory 116 of client device 110 and executing on processor 112 of client device 110. In some embodiments, any number of instances of any number of software applications may reside in memory 116 and any number of other memories associated with any number of other computing instances, and execute in any combination on processor 112 of client device 110 and any number of other processors associated with any number of other computing instances. In the same or other embodiments, the functionality of any number of software applications may be distributed across any number of other software applications residing in memory 116 and any number of other memories associated with any number of other computing instances, and execute in any combination on processor 112 and any number of other processors associated with any number of other computing instances. Furthermore, a subset of the functionality of multiple software applications may be merged into a single software application.

[0019] Specifically, client device 110 is configured to implement design exploration application 130 to generate one or more two-dimensional (2D) or 3D designs, such as 2D floor plan designs and / or 3D designs for 3D objects. In some embodiments, design exploration application 130 causes one or more ML models 180, 190 to synthesize designs of 3D objects based on any number of objectives and constraints. Design exploration application 130 then presents the designs to the user as one or more design objects 144 within a design space context. In some embodiments, design objects 144 include 2D objects, such as sub-parts of a 2D design, each sub-part including 2D geometry. For example, a 2D design may include a building layout, and 2D design objects 144 may include specific rooms within the building layout. The embodiments and techniques described herein can handle both 3D and 2D designs and design objects 144 in a similar manner. In some embodiments, a user can explore and modify 2D or 3D design objects 144 via GUI 120.

[0020] In various embodiments, processor 112 can be any instruction execution system, device, or apparatus capable of executing instructions. For example, processor 112 may include a general-purpose processor (such as a central processing unit), a special-purpose processor (such as a graphics processing unit), a special-purpose processor, a field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of different processing units. In some embodiments, processor 112 is a programmable processor that executes program instructions to manipulate input data. In some embodiments, processor 112 may include any number of processing cores, memory, and other modules for facilitating program execution.

[0021] Input / output (I / O) device 114 includes means configured to receive input, such as a keyboard, mouse, trackball, etc. In some embodiments, I / O device 114 also includes means configured to provide output, such as a display device, speaker, etc. For example, the input means may enable a user to control a cursor displayed on the output means to select various elements displayed on the output means 114. Additionally or alternatively, I / O device 114 may further include means configured to receive and provide input and output respectively, such as a touchscreen, a universal serial bus (USB) port, etc.

[0022] Memory 116 includes storage modules or a collection of storage modules. In some embodiments, memory 116 may include various computer-readable media selected based on their size, relative performance, or other capabilities: volatile and / or non-volatile media, removable and / or non-removable media, etc. Memory 116 may include cache, random access memory (RAM), memory, etc. Memory 116 may include one or more discrete memory modules, such as dynamic RAM (DRAM) dual in-line memory modules (DIMMs). Of course, various memory chips, bandwidths, and form factors may also be selected. Memory 116 stores content (such as software applications and data) used by processor 112. In some embodiments, memory (not shown) supplements or replaces memory 116. Memory may include any number and type of external memory accessible to the processor 112 of client device 110. For example, but not limited to, storage devices may include secure digital (SD) cards, external flash memory, portable optical disc read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0023] The non-volatile memory included in memory 116 typically stores one or more applications (including design exploration application 130) and data for processing by processor 112. For exampleThis includes data files 142 and / or design objects stored in local data storage 140. In various embodiments, memory 116 may include non-volatile memory, such as an optical disc drive, magnetic drive, flash memory, or other memory. In some embodiments, separate data storage (such as one or more external data storages connected via network 150 (“cloud storage”)) may supplement memory 116. In various embodiments, design exploration application 130 within memory 116 may be executed by processor 112 to implement the overall functionality of client device 110, thereby coordinating the operation of system 100 as a whole.

[0024] In various embodiments, memory 116 may include one or more modules for performing the various functions or techniques described herein. In some embodiments, one or more of the modules and / or applications included in memory 116 may be implemented locally on client device 110 and / or via a cloud-based architecture. For example, any of the modules and / or applications included in memory 116 may be implemented on a remote device communicating with client device 110 via a network interface or I / O device interface. For example It can be executed on smartphones, server systems, cloud computing platforms, etc.

[0025] Design exploration application 130 resides in memory 116 and executes on processor 112 of client device 110. Design exploration application 130 interacts with the user via GUI 120. In various embodiments, design exploration application 130 operates as a 2D or 3D design application to generate and modify an overall 2D or 3D design including one or more 2D or 3D design objects 144. Design exploration application 130 interacts with the user via GUI 120 to allow direct user input (…). For example The design explores the application of one or more tools (130) to generate 3D objects, wireframe geometry, meshes, etc., or via a separate device. For example One or more design objects 144 are generated by a trained ML model 180, a remote ML model 190, or a standalone 3D design application. When one or more design objects 144 are generated via a standalone device, the design exploration application 130 uses one or more modalities ( For exampleThe design exploration application 130 generates prompts (based on user input, including text, voice, images, etc.) that effectively describe the design-related intent. Then, the design exploration application 130 causes one or more of the ML models 180, 190 to act on the generated prompts to produce relevant ML responses, such as relevant design objects 144. The design exploration application 130 receives ML responses (such as design objects 144) from one or more ML models 180, 190 and displays the ML responses (such as design objects 144) within the GUI 120. The user can select design objects 144 via the GUI 120 for modification or use, such as incorporating design objects 144 into a larger overall 3D design displayed in the GUI 120.

[0026] GUI 120 can be any type of user interface that allows a user to interact with one or more software applications via any number and / or type of GUI elements. GUI 120 can be displayed in any technically feasible manner on any number and / or type of standalone display devices, any number and / or type of displays integrated into any number and / or type of user devices, or any combination thereof. Design exploration application 130 can perform any number and / or type of operations to directly and / or indirectly display and monitor any number and / or type of interactive GUI elements and / or any number and / or type of non-interactive GUI elements within GUI 120. In some embodiments, each interactive GUI element implements one or more types of user interactions that automatically trigger corresponding user events. Some examples of interactive GUI element types include, but are not limited to, scrollbars, buttons, text input boxes, drop-down lists, and sliders. In some embodiments, design exploration application 130 organizes GUI elements into one or more container GUI elements (...). For example (panes and / or panes).

[0027] Local data storage 140 is part of the memory in client device 110, storing one or more design objects 144 included in the overall 3D design and / or one or more data files 142 associated with the overall 3D design. For example, the overall 3D design of a building may include multiple stored design objects 144, including design objects 144 representing doors, windows, fixtures, walls, appliances, etc. Local data storage 140 may also include data files 142 associated with the overall 3D design. For example (Component files, metadata, etc.). Additionally or alternatively, local data storage 140 includes data files 142 associated with generating prompts for transfer to one or more ML models 180, 190. For example, local data storage 140 may store data for sketches, geometry, etc. For example (wireframes, meshes, etc.), images, videos, application status ( For example142. One or more data files, such as camera angles used within the design space, tools selected by the user, audio recordings, etc.

[0028] Design object 144 includes geometry, texture, images, and / or other components used by design exploration application 130 to generate an overall 2D or 3D design. In some embodiments, the geometry of a given design object refers to any multidimensional model of the physical structure, including CAD models, meshes, and point clouds, as well as building layouts, circuit layouts, piping diagrams, freeform diagrams, etc. In some embodiments, design exploration application 130 stores multiple design objects 144 for a given overall 3D design and stores multiple iterations of a given target object that have been iteratively modified by ML modeling 180, 190. For example, a user can use design exploration application 130 to form and input a first prompt and receive a first generated design object 144 from trained ML model 180 as an ML response to the first prompt, and then ( For example (by refining the first prompt) submitting the second prompt and receiving the second generated design object 144 from the trained ML model 180 as an ML response to the second prompt.

[0029] Network 150 can be any technically feasible set of interconnected communication links, including a local area network (LAN), a wide area network (WAN), the World Wide Web, or the Internet. Network 150 enables communication between client device 110 and other devices in network 150 via wired and / or wireless communication protocols, including Bluetooth, Bluetooth Low Energy (BLE), Wi-Fi, cellular protocols, satellite networks, and / or Near Field Communication (NFC).

[0030] Server device 160 is configured to communicate with design exploration application 130 to generate one or more ML responses (such as design objects) in response to one or more prompts. In operation, server device 160 executes intent management application 170 to process received prompts generated by design exploration application 130, selects one or more ML models 180, 190 trained to generate design object 144 in response to the content of the prompt (based on the received prompt), and inputs the prompt into the selected ML models 180, 190. Once the selected ML models 180, 190 generate design object 144 in response to the prompt, server device 160 transmits the generated design object to client device 110, whereby the generated design object 144 can be used by design exploration application 130. For example, design exploration application 130 may display the generated design object 144 in GUI 120 for the user to explore, manipulate, and / or modify.

[0031] In various embodiments, processor 162 can be any instruction execution system, device, or apparatus capable of executing instructions. For example, processor 162 may include a central processing unit (CPU), digital signal processing unit (DSP), microprocessor, application-specific integrated circuit (ASIC), neural processing unit (NPU), graphics processing unit (GPU), field-programmable gate array (FPGA), controller, microcontroller, state machine, or any combination thereof. In some embodiments, processor 162 is a programmable processor that executes program instructions to manipulate input data. In some embodiments, processor 162 may include any number of processing cores, memory, and other modules for facilitating program execution.

[0032] Input / output (I / O) device 164 includes means configured to receive input, such as a keyboard, mouse, etc. In some embodiments, I / O device 164 further includes means configured to provide output, such as a display device, speaker, etc. Additionally or alternatively, I / O device 164 may further include means configured to receive and provide input and output, such as a touchscreen, Universal Serial Bus (USB) port, etc.

[0033] Memory 166 includes storage modules or a collection of storage modules. In some embodiments, memory 166 may include various computer-readable media selected based on their size, relative performance, or other capabilities: volatile and / or non-volatile media, removable and / or non-removable media, etc. Memory 166 may include cache, random access memory (RAM), memory, etc. Memory 166 may include one or more discrete memory modules, such as dynamic RAM (DRAM) dual in-line memory modules (DIMMs). Of course, various memory chips, bandwidths, and form factors may also be selected. Memory 166 stores content (such as software applications and data) used by processor 162. In some embodiments, memory (not shown) supplements or replaces memory 166. Memory may include any number and type of external memory accessible to the processor 162 of server device 160. For example, but not limited to, storage devices may include secure digital (SD) cards, external flash memory, portable optical disc read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0034] The non-volatile memory included in memory 166 typically stores one or more applications (including intent management application 170 and one or more trained ML models 180) and data for processing by processor 112. For example(Design History 182). In various embodiments, memory 166 may include non-volatile memory, such as an optical disc drive, magnetic drive, flash memory drive, or other memory. In some embodiments, separate data storage (such as one or more external data storages connected via network 150) may supplement memory 166. In various embodiments, intent management application 170 and / or one or more ML models 180 within memory 166 may be executed by processor 162 to implement the overall functionality of server device 160, thereby coordinating the operation of system 100 as a whole.

[0035] In various embodiments, memory 166 may include one or more modules for performing the various functions or techniques described herein. In some embodiments, one or more of the modules and / or applications included in memory 166 may be implemented locally on client device 110, server device 160, and / or via a cloud-based architecture. For example, any of the modules and / or applications included in memory 166 may be implemented on a remote device communicating with server device 160 via a network interface or I / O device interface. For example It can be executed on smartphones, server systems, cloud computing platforms, etc. Additionally or alternatively, the intent management application 170 can be executed on client device 110 and can communicate with a trained ML model 180 operating at server device 160.

[0036] In various embodiments, the intent management application 170 receives a prompt from the design exploration application 130 and selects and inputs the prompt into the applicable ML models 180, 190. In some embodiments, one or more of the ML models 180, 190 are trained to respond to specific types of input, such as being trained to select from specific modal combinations (…). For example (Text and images) to generate an ML model of the design object. In this case, the intent management application 170 processes the prompt to determine the modality including and / or specifying the data in the prompt, and identifies one or more ML models 180, 190 that have been trained to respond to such a combination of modalities. When identifying one or more appropriate ML models, the intent management application 170 selects an ML model (text and images) to generate an ML model of the design object. For example (The trained ML model 180) and inputs the prompt into the selected ML model 180.

[0037] The trained ML model 180 includes one or more generative ML models, which have been trained on a relatively large amount of existing data and any number of possible results. For exampleThe ML model 180 is trained on design objects 144 and user-provided evaluations to perform any number and / or type of prediction tasks based on patterns detected in existing data. In various embodiments, a remote ML model 190 is an additional trained ML model that communicates with server device 160 to receive cues via intent management application 170. In some embodiments, the trained ML model 180 is trained using various combinations of data from multiple modalities, such as text data, image data, audio data, etc. For example, in some embodiments, one or more trained ML models 180 may include a third-generation generative pre-trained transformer (GPT-3) model, a specialized version of the GPT-3 model known as the “DALL-E2” model, a fourth-generation generative pre-trained transformer (GPT-4) model, etc. In various embodiments, the trained ML model 180 may be trained to generate design objects from various combinations of modalities. These combinations include text, CAD objects, geometry, images, sketches, videos, application states, audio recordings, etc.

[0038] In some embodiments, user-provided evaluations can be used to retrain ML models 180 and 190. In these embodiments, a user can submit a first prompt for a specific design object 144 via a design exploration application 130, which submits the first prompt to an intent management application 170. The intent management application 170 then inputs the first prompt into a selected ML model, which generates the design object 144 in response to the first prompt. The generated design object 144 is then returned to the design exploration application 130, which displays the generated design object 144 in a GUI 120. The user can then enter a second prompt, which includes an evaluation (feedback) of the response received to the first prompt (i.e., the generated design object 144), such as "the received object is correct" or "the received object is incorrect." The intent management application 170 can then store the first prompt, the generated design object 144, and the second prompt as additional training data for retraining ML models 180 and 190 at a later time to improve their accuracy. As discussed below, additional training data for retraining can be stored in design history 182.

[0039] Design history 182 includes data and metadata associated with one or more trained ML models 180 and / or one or more remote ML models 190 that generate design objects 144 in response to prompts provided by design exploration application 130. In some embodiments, design history 182 includes successive iterations of design objects 144 generated by a single ML model 180 in response to a series of prompts. Additionally or alternatively, design history 182 includes multiple design objects 144 generated by different ML models 180, 190 in response to the same prompt. In some embodiments, design history 182 includes evaluation feedback provided by the user for a given design object 144. In this case, server device 160 can use design history 182 as additional training data to retrain one or more ML models 180. Additionally or alternatively, design exploration application 130 can retrieve the contents of design history 182 and display the retrieved contents to the user via GUI 120.

[0040] Figure 2 According to various embodiments Figure 1 A more detailed illustration of the design exploration application 130 is shown. As illustrated, in some embodiments, system 200 includes, but is not limited to, design exploration application 130, GUI 120, local data storage 140, one or more data files 142, context data 146, one or more remote context databases 194, server device 160, remote ML model 190, and prompts 260.

[0041] GUI 120 includes, but is not limited to, a cue space 220, zero or more cue history markers 222, and a design space 230. Design exploration application 130 includes, but is not limited to, an intent manager 240 including one or more keyword datasets 242, one or more design objects 144, a visualization module 250, and one or more cue history tables 252. Server device 160 includes, but is not limited to, an intent management application 170, one or more trained models 180, a design history 182, and one or more ML responses 280 (including one or more design objects 270) generated in response to received cue 260. Cue 260 includes, but is not limited to, design intent text 262, one or more design files 264, one or more design space references 266, and / or context information 268.

[0042] For illustrative purposes only, the functionality of the design exploration application 130 is described herein within the context of an exemplary interactive and linear workflow used to generate the generated design object 270 based on user-based design-related intent expressed during the workflow. The generated design object 270 includes, but is not limited to, one or more images, wireframe models, geometries, and / or meshes for 2D or 3D design, and any amount (including none) and / or type of associated metadata.

[0043] As those skilled in the art will recognize, the techniques described herein are illustrative and not limiting, and can be modified and applied in other contexts without departing from the broader spirit and scope of the inventive concepts described herein. For example, during the entire process of generating and evaluating a design for the target 3D object, the techniques described herein can be modified and applied to generate any number of generated design objects 270 associated with any target 3D object in a linear, nonlinear, iterative, non-iterative, recursive, non-recursive manner, or any combination thereof. The target 3D object may include any number (including one) and / or type of target 3D objects and / or target 3D object components.

[0044] For example, in some embodiments, a generated design object 270 may be generated and displayed within GUI 120 during a first iteration, any portion (including all) of the design object 270 may be selected via GUI 120, and a first prompt 260 may be set equal to the selected portion of the generated design object 270 to recursively generate a second generated design object 270 during a second iteration. In the same or other embodiments, when generating each newly generated design object 270, the design exploration application 130 may display and / or re-display any number of GUI elements, generate and / or regenerate any number of data, or any combination thereof, any number and / or in any order.

[0045] In operation, the visualization module 250 of the design exploration application 130 generates and renders a GUI 120, which includes a cue space 220, zero or more cue history markers 222, and a design space 230. A user can provide at least a portion of the content for a cue 260 via the cue space 220 (by entering user input in the cue space 220). The design exploration application 130 processes the content entered into the cue space 220 to generate the cue 260, adds optional additional information (such as context information 268), and transmits the cue 260 to a server device 160. The intent management application 170 identifies the modality of the data included in the cue 260 and selects one or more trained ML models 180 and / or remote ML models 190 that have been trained to process the identified modality. The intent management application 170 inputs the cue into one or more of the selected ML models 180, 190. ML models 180 and 190 respond to cue 260 by generating one or more ML responses 280, which may include one or more design objects 270 and additional information, such as one or more text responses. Visualization module 250 receives one or more generated design objects 270 and displays one or more generated design objects 270 in cue space 220 and / or design space 230.

[0046] In various embodiments, design space 230 is a virtual workspace that includes one or more renderings of design objects that form the overall 3D design. For example (The geometry of the current design object 144 and / or the newly generated design object 270). In some embodiments, the design space includes multiple design alternatives for the overall 3D design. For example, the design space 230 may graphically organize multiple 3D designs including different combinations of design objects 144, 270. In such cases, the design exploration application 130 provides various tools that enable users to interact with the GUI 120 to explore the design space 230 and navigate between design alternatives to quickly analyze trade-offs between different design options, observe trends in design options, constrain the design space, select specific design options, and so on.

[0047] The prompt space 220 is a panel where a user can input content to generate the prompt 260. In various embodiments, the intent manager 240 determines the intent of the input provided by the user. For example, the intent manager 240 may include a natural language (NL) processor that parses the text provided by the user. Additionally or alternatively, the intent manager 240 may include processing audio data to identify words included in the audio data and parsing the identified words. In various embodiments, the intent manager 240 identifies one or more keywords in the text data. In some embodiments, the intent manager 240 includes one or more keyword datasets 242 that the intent manager 240 references when identifying one or more keywords included in the text data.

[0048] For example, keyword dataset 242 may include, but is not limited to, a 3D keyword dataset containing any number and / or type of 3D keywords, a custom keyword dataset containing any number and / or type of custom keywords, and / or a user keyword dataset containing any number and / or type of user keywords. For example A user-specified dataset of words and / or phrases. Keywords may include specific words or phrases related to the design of 3D objects. For example (Indicative pronouns, technical terms, reference terms, etc.). For example, a user can enter a regular sentence ("I want a hinge to connect here") in the input area within cue space 220. The intent manager identifies "hinge," "connect," and "here" as words related to the ML models 180 and 190 that generate the design object 270. In this case, intent manager 240 can update cue space 220 by highlighting keywords, allowing the user to provide additional details (indicative pronouns, technical terms, reference terms, etc.). For example (Non-text data) to be included in prompt 260.

[0049] The design exploration application 130 receives text and / or non-text data via an input area included in the prompt space 220 for inclusion in the prompt 260. When non-text data is provided, the user can retrieve stored data from local data storage 140, such as one or more stored data files 142. For example (This includes stored geometry, stored CAD files, audio recordings, stored sketches, etc.). Additionally or alternatively, the user can retrieve content from the design history 182 and add content to the input area. In this case, the content from the design history 182 is stored in one or more data files 142, which the user retrieves from the local data storage 140.

[0050] Hint 260 is a hint specifying the user's design intent. In various embodiments, design exploration application 130 receives content / data and constructs hint 260 based on the received content / data. For example, the user may initially write design intent text 262 involving a sketch. Then, design exploration application 130 receives the sketch ( For example (This could be a stored sketch or a sketch entered by the user into the input design area). Upon receiving a sketch, the design exploration application 130 can then generate a prompt 260 to include both design intent text 262 and the sketch. The design intent text 262 includes textual data describing the user's intent. For example, the design intent text may include information about a target 3D design object (…). For example The description of the property is "a handle made of titanium". In some embodiments, the design exploration application 130 generates design intent text from different types of data input. For example, the intent manager 240 can perform NL processing to identify words included in an audio recording. In this case, the design exploration application 130 generates design intent text 262 including the identified words.

[0051] Design file 264 includes one or more files that the user adds to be included in prompt 260. For example CAD files, stored text, audio recordings, stored geometry, etc.). In some embodiments, design file 264 may include text data ( For example (Text descriptions, physical dimensions, etc.). In some embodiments, the design exploration application 130 converts various types of data into a design file 264. For example, a user may record audio via an input area. In this case, the design exploration application 130 may store the audio recording as a design file 264. The design file 264 may include one or more modalities (…). For example (Text data, video data, audio data, image data, etc.)

[0052] In some embodiments, design space reference 266 may include one or more references to prompt space 220 and / or design space 230. For example, a user may enter text referencing a specific application state ( For example Examples include phrases like "make the thing selected by the current tool lighter" and "generate seat for the car in this view." In this case, the design exploration application 130 determines the application state that the user is referencing. The design exploration application 130 can then include this reference as a design space reference 266 in the prompt 260.

[0053] In various embodiments, the intent management application 170 receives and processes the prompt 260 to identify the modality of the content of the prompt 260. For example, the intent management application 170 processes modalities including design intent text 262, one or more design files 264, and / or one or more design space references 266 included in the prompt 260. For example, the intent management application 170 may identify combinations of text, image, and video modalities included in the prompt. The intent management application 170 identifies at least one ML model 180, 190 trained with the modal combination and selects one of the identified ML models 180, 190. The intent management application 170 executes the selected ML model by inputting the prompt 260 into the selected ML model. The selected ML model generates a design object 270 in response to the prompt 260. In some embodiments, the server device 160 includes the generated design object 270 in a design history 182. In this case, the generated design object 270 is part of the design history 182 and can be used as additional training data to retrain one or more trained ML models 180. For example (Further training the selected ML model, training other ML models, etc.).

[0054] Figure 3 According to various embodiments Figure 2A conceptual diagram of one of the prompt history tables 252. In some embodiments, the prompt history table 252 includes any type of data structure or data container for storing and organizing data. When the design exploration application 130 first receives user input via the prompt space 220 for a specific design object 144 or empty location within the design space 230, it can generate a prompt history table 252 and store it in memory 116 for that specific design object 144 or empty location. Thus, each prompt history table 252 is generated to store the prompt history for the associated design object 144 or empty location within the design space 230. In some embodiments, each prompt history table 252 is a separate and independent table associated with a different design object 144 or different empty location within the design space 230. In this way, the design exploration application 130 can separately store and track multiple different prompt histories for multiple different design objects 144 and / or empty locations within the design space 230. Please note that when a user interacts with an empty location / space within design space 230 (i.e., a location / space without design object 144), for example, to generate design object 144 in the empty location / space, but changes his / her mind and decides not to generate any design object 144 in the empty location / space, a prompt history table 252 may be generated for that empty location / space. Such user interaction with the empty location / space may include user input and corresponding ML responses 280 in prompt space 220, which are recorded in the prompt history table 252 generated and associated with that empty location / space.

[0055] As shown in the figure, the prompt history table 252 includes a first section 310 and a second section 350. The first section 310 stores data describing the prompt history table 252 and the associated design object 144 or empty location within the design space 230. The second section 350 stores prompt interactions received in the prompt space 220 for the associated design object 144 or empty location, including user input and corresponding ML responses 280.

[0056] As shown in the figure, the first segment 310 includes a table type field 320, an identifier field 330, and a count field 340. The table type field 320 specifies the type of design item associated with the cue history table 252, such as an assembly object, part object, component object, or empty location. As discussed below, the table type field 320 can be used to identify which cue history tables 252 and associated design objects 1144 correspond to the current zoom level of the design space 230, so that specific cue history markers 222 are displayed only within the design space 230 based on the current zoom level. The identifier field 330 uniquely identifies the cue history table 252 and the design items associated with it. In these embodiments, the design exploration application 130 stores a unique identifier for each design object 144 within the design space 230. Therefore, for a design item that includes a design object 144, the identifier field 330 can specify a unique identifier for the design object 144. For a design item that includes an empty location, the identifier field 330 can specify the unique xyz coordinates of the empty location within the design space 230. Please note that when multiple cue history tables 252 are generated and stored, the identifier field 330 also uniquely identifies each cue history table 252 within the multiple cue history tables 252. The count field 340 stores a value used for cue interaction counting, which includes the total number of cue interactions received so far in the cue space 220 for the associated design object 144 or empty location. The cue interaction count can be incremented for each user input or ML response 280 received in the cue space 220.

[0057] As shown in the figure, the second segment 350 includes multiple user input entries 360 (such as 360a, 360b, etc.) and multiple ML response entries 370 (such as 370a, 370b, etc.). Each user input entry 360 stores data associated with a specific user input received in the prompt space 220 for the associated design object 144 or an empty location. For example, the user input entry 360 for user input may include user text input (e.g., natural language), the camera view (camera position and camera orientation) of the associated design object 144 recorded when the user input is received in the prompt space 220, and / or a link to a snapshot image captured for the associated design object 144 when the user input is received in the prompt space 220. Each ML response entry 370 stores data associated with a specific ML response 280 received in the prompt space 220 for the associated design object 144 or an empty location, such as a text response (e.g., natural language) to a prompt 260 generated for the corresponding user input.

[0058] Implementation Examples Overview In operation, the visualization module 250 updates the prompt space 220 and / or design space 230 based on user input and / or data (response) received from the server device 160. Initially, the visualization module 250 may display the design space 230 without displaying the prompt space 220, allowing the user to explore and navigate the design space 230 without visual clutter. For example, the user can move the cursor to select various design objects 144 and / or empty spaces within the design space 230. If the user expects to generate prompts 260 to receive design guidance or assistance from the ML models 180, 190, the user can invoke the prompt space 220 via a predetermined hotkey input.

[0059] Typically, the prompt space 220 is a panel, such as a window, separate from the design space 230. In some embodiments, the prompt space 220 is a panel / window integrated within / inside the design space 230, whereby at least a portion of the prompt space 220 overlaps / covers at least a portion of the design space 230. In response to a user's call to the prompt space 220, the visualization module 250 may place / position the prompt space 220 within the design space 230 in a manner that facilitates and improves the user's focus on the currently selected location in the design space 230. As described above, in conventional methods, the prompt space 220 is displayed as a panel separate from and outside the design space 230 in a way that distracts the user's focus on the currently selected location in the design space 230.

[0060] In some embodiments, the visualization module 250 displays a prompt space 220 at the current position of the user-controlled cursor or the currently selected position within the design space 230. For example, the currently selected position may be on the selected design object 144, so the visualization module 250 can display the prompt space 220 close to / adjacent to the selected design object 144 within the design space 230. For example, the currently selected position may be an empty position within the design space 230, so the visualization module 250 can display the prompt space 220 close to / adjacent to the selected empty position within the design space 230. The prompt space 220 is used to interact with the ML models 180, 190 by submitting a prompt 260 and then displaying the ML model response 280 to the prompt 260. The prompt space 220 includes an input area where the user can enter the content / data (user input) included in the prompt 260. The prompt space 220 also includes a response area that displays the ML response 280 to the prompt 260. For example, ML response 280 may include design object 270 and / or text response, whereby the received design object 270 may be displayed in design space 203 and / or prompt space 220, and the received text response may be displayed in prompt space 220. Prompt space 220 may be used to perform one or more iterations of: (1) entering user input / submit prompt 260 and (2) receiving ML response 280. When the user no longer wishes to use prompt space 220, the user may close prompt space 220 via a predetermined hotkey, which will cause prompt space 220 to no longer be displayed in design space 230 and / or GUI 120.

[0061] Please note that after one or more iterations of (1) entering user input / submitting prompt 260 and (2) receiving ML response 280, prompt space 220 effectively displays prompt history, which includes all prompt interactions / messages received within prompt space 220 for the selected design object 144 or the selected empty location. The prompt interactions in the prompt history include one or more entered user inputs associated with one or more prompts 260, and one or more corresponding ML responses 280 (such as one or more text responses) received for one or more prompts 260. In some embodiments, prompt history is associated with a specific design object 144 or a specific empty location within design space 230. In these embodiments, the prompt history associated with a specific design object 144 or a specific empty location includes only those prompt interactions received for that specific design object 144 or a specific empty location, and excludes prompt interactions received for other design objects 144 or empty locations. Visualization module 250 stores and maintains one or more prompt histories associated with one or more design objects 144 and / or one or more empty locations within design space 230 via one or more prompt history tables 252. For multiple prompt history tables 252, each prompt history table 252 is a separate and independent table associated with different design objects 144 or different empty locations within the design space 230. In this way, the visualization module 250 can separately store and track multiple different prompt histories for multiple different design objects 144 and / or empty locations within the design space 230.

[0062] In some embodiments, the visualization module 250 may also display zero or more cue history markers 222 within the design space 230. Each cue history marker 222 represents a specific cue history table 252 and cue history associated with a specific design object 144 or empty location within the design space 230. The cue history marker 222 may also display a count value for the total number of cue interactions included in the corresponding cue history. The visualization module 250 may display the cue history marker 222 close to / adjacent to the corresponding design object 144 or empty location within the design space 230 to indicate to the user that there is a previous cue history associated with the specific design object 144 or empty location that the user may wish to explore and review. The user may then select a cue history marker 222, and in response, the visualization module 250 retrieves the corresponding cue history table 252 and displays the corresponding cue history associated with the specific design object 144 or empty location.

[0063] To display the prompt history, the visualization module 250 can open and display the prompt space 220, which is placed near the associated design object 144 or empty location. In some embodiments, the visualization module 250 only displays the prompt history associated with a specific design object 144 or empty location, and does not display the prompt history not associated with a specific design object 144 or empty location. Through the prompt space 220, the user can view the relevant prompt history and can also enter new user input to generate new prompts 260, which are then added to the prompt history associated with the specific design object 144 or empty location and the prompt history table 252.

[0064] In some embodiments, the design exploration application 130 provides a zoom tool that enables different zoom levels for viewing the design space 230. Furthermore, the design objects 144 in the design space 230 may include assemblies, parts, or components. In these embodiments, the visualization module 250 displays specific cue history markers 222 based on the current zoom level selected for viewing the design space 230, thereby displaying only those cue history markers 222 that are appropriate for the current zoom level and excluding those that are not. For example, if a first current zoom level includes a low “zoom-out” level corresponding to assembly-level zoom, the visualization module 250 displays only those cue history markers 222 associated with the design object 144 that includes the assembly, and excluding those associated with the design object 144 that includes the part or component. For example, if a second current zoom level includes a medium “neutral zoom” level corresponding to part-level zoom, the visualization module 250 displays only those cue history markers 222 associated with the design object 144 that includes the part, and excluding those associated with the design object 144 that includes the assembly or component. For example, if the third current zoom level includes a high “zoom” level corresponding to the component-level zoom, the visualization module 250 only displays those prompt history markers 222 associated with the design object 144 that includes the component, and not the prompt history markers 222 associated with the design object 144 that includes the assembly or part.

[0065] In some embodiments, for each user input entered into the prompt space 220 and associated with the design object 144, the visualization module 250 also captures the user / camera viewpoint of the design object 144 at the point in time the user input was entered into the prompt space 220. The captured user / camera viewpoint is then stored in a corresponding prompt history table 252, which represents the prompt history associated with the design object 144, thereby associating the captured viewpoint with a specific user input within the prompt history. Later, when the user views the prompt history associated with the design object 144, the user can select a specific user input from the prompt history, and in response, the visualization module 250 can apply the captured viewpoint to the design object 144 within the design space 230. In this way, the user can see the viewpoint of the design object 144 at the moment the user entered the corresponding user input, providing context and a better understanding of that user input.

[0066] In some embodiments, for each user input entered into prompt space 220 that relates to a specific design object 144 or empty location within design space 230, intent manager 240 automatically retrieves additional context information 268 and includes the additional context information 268 in prompt 260 generated for that user input. The additional context information 268 can be retrieved from context data 146 stored in local data storage 140. Context data 146 may include data collected and stored by design exploration application 130 during the execution of the design application to design design space 230. The additional context information 268 can be retrieved from one or more remote context databases 194. For example, remote context database 194 may include a database for manufacturers of assemblies and / or parts, containing information for assemblies and / or parts. The additional context information 268 may be included in prompt 260 to provide context information to user input, allowing ML models 180, 190 to provide a more accurate and useful ML model response 280 to prompt 260.

[0067] Placement of prompts within the design space Initially, the design exploration application 130 generates a GUI 120, which includes a design space 230 comprising zero or more design objects 144. Design objects 144 may include assemblies, parts, or components. Assemblies include design objects 144 that comprise multiple connected but distinct parts, each part comprising a separate design object 144. Parts include design objects 144 that comprise multiple connected but distinct elements (such as edges and / or faces), each element comprising a separate design object 144. Thus, an assembly object includes multiple sub-objects, which include part objects, and a part object includes multiple sub-objects, which include component objects. The design exploration application 130 provides various tools for selecting, manipulating, and modifying assembly objects, part objects, or component objects. For example, the design exploration application 130 provides assembly selection tools, part selection tools, and component selection tools. As another example, the design exploration application 130 provides zoom tools for viewing the design space 230, enabling zooming out to view and select assemblies, zooming in on the view of assemblies to view and select specific parts of the assembly, and zooming in further on the view of the parts to view and select specific elements of the parts. In the embodiments described below, various innovative functions and processes are performed on the design object 144, which includes part objects. However, in other embodiments, the various innovative functions and processes described below can be similarly performed on the design object 144, which includes assembly objects or element objects. In further embodiments, the various innovative functions and processes described below can also be similarly performed on selected empty locations within the design space 230.

[0068] Figures 4-8 A cue placement feature is illustrated, which places a cue space 220 within the design space 230 based on user interaction within the design space 230. The cue placement feature allows the user to invoke the cue space 220 from any location within the design space 230. In some embodiments, various innovative functions and processes for implementing the cue placement feature are performed by the design exploration application 130 and its various modules, such as... Figure 2 As shown.

[0069] Figure 4 According to various embodiments Figure 2An exemplary illustration of the initial selection of a design object 144 within the design space 230. As shown, the design space 230 includes a first design object 410, which comprises part objects, thereby allowing the user to select the first design object 410 by moving the user control cursor 430 to a specific location within the design space 230. In response, the design exploration application 130 displays a collapsed prompt space 420 within the design space 230, positioned close to the currently selected location and the currently selected first design object 410. The collapsed prompt space 420 displays a pointed symbol to indicate that no prior prompting interaction has occurred for the selected first design object 410.

[0070] Generally, various techniques can be used to place a first GUI item "near" or "adjacent" to a second GUI item within design space 230. In some embodiments, various techniques can be used to place the prompt space 220 near the currently selected location or currently selected design object 144 within design space 230. For example, the center point of prompt space 220 can be placed within a predetermined distance from the currently selected location or design object 144 within design space 230. Alternatively, the side of prompt space 220 closest to the currently selected location or design object 144 within design space 230 can be placed within a predetermined distance from the currently selected location or design object 144 within design space 230. For example, the predetermined distance can be measured in pixels or the native coordinate system of design space 230. In other embodiments, other techniques are used to place prompt space 220 near the currently selected location or currently selected design object 144 within design space 230. For example, another approach is to place the prompt space 220 in the middle between the currently selected location or design object 144 and the nearest edge of the screen, or to place the prompt space 220 at the same point on the screen, and to include a "guideline" connecting the prompt space 220 to the currently selected location or design object 144. For example, similar techniques can be used to place the prompt history marker 222 near the associated design object 144 or an empty space within the design space 230, or to place a snapshot image near the associated user input within the prompt space 220.

[0071] Figure 5 According to various embodiments Figure 4 An exemplary illustration of the interaction mode of the prompt space within the design space 230. (Refer to...) Figure 4After selecting the first design object 410, the user can then begin a new prompt interaction by invoking / activating the interaction mode of the prompt space. For example, the interaction mode of the prompt space can be invoked by selecting the collapsed prompt space 420, entering a predetermined hotkey combination, or entering a specific voice command. In some embodiments, when in interaction mode, the cursor cannot move and remains stationary at its current position within the design space 230, and can move again when the interaction mode is disabled.

[0072] In response to the activation of the interaction mode, the design exploration application 130 displays an extended tooltip space 520 within the design space 230, positioned near the currently selected location and the currently selected first design object 410. The extended tooltip space 520 includes an input area (indicated by the ">" symbol) where the user can enter text input to generate tooltips 260 to be sent to ML models 180 and 190. Figure 5 As shown in the example, the user enters the query "How could I make this lighter?" with reference to the currently selected first design object 410.

[0073] In some embodiments, upon receiving a first prompt interaction for a selected design object 144 or empty location within the design space 230, the design exploration application 130 generates and stores a new prompt history table 252 associated with the selected design object 144 or empty location. (As mentioned above regarding...) Figure 3 As discussed, the design exploration application 130 captures and stores various information in a prompt history table 252 to record the prompt history associated with the selected design object 144 or an empty location. For example, for each user input associated with the selected design object 144, the design exploration application 130 may capture the camera viewpoint of the selected design object 144 at the time the user input is entered into the prompt space 220. The user can then review the prompt history for a particular design object 144 or empty location later, as discussed below.

[0074] Figure 6 According to various embodiments Figure 5 An exemplary illustration of ML response 280 shown in the prompt space. (See reference...) Figure 5The user enters user input into the prompt space 520, and in response, the design exploration application 130 generates a prompt 260 based on the user input and sends the prompt 260 to the server 160. In addition to user input, the design exploration application 130 may include further data in the generated prompt 260, such as design intent text 262, one or more design files 264, one or more design space references 266, and / or context information 268. Specifically, when the user references the selected first design object 410 in the query "How could I make this lighter?", the design exploration application 130 may include the geometry of the selected first design object 410 in the generated prompt 260. The server 160 (referred to as "Advisor") implements ML models 180, 190 and returns an ML response 280 to the prompt 260. Figure 6 As shown in the example, the ML response 280 from "Advisor" includes a natural language text response to the user's input query ("You could switch the material from steel to titanium.").

[0075] Figure 7 According to various embodiments Figure 5 An exemplary illustration of additional user input shown in the prompt space. (See reference) Figure 6 ML response 280 suggests that the user change the material of the selected first design object 410 from steel to titanium. In response, the user confirms the proposed material change to titanium by entering the text "OK, let's try that" in the input area of ​​prompt space 520. Note that when further prompting interactions (user input and ML response 280) are received for the selected first design object 410, prompt space 520 automatically scrolls to the last prompting interaction, but the user can scroll up to view previous prompting interactions.

[0076] Figure 8 According to various embodiments Figure 5 An exemplary illustration of the additional ML response shown in the prompt space. (See reference) Figure 7 In response to user input confirming the material change to titanium, the design exploration application 130 generates additional prompt 260 based on the user input and sends the additional prompt 260 to server 160. Server 160 implements ML models 180 and 190 and returns an additional ML response 280 for the additional prompt 260. Figure 7As shown in the example, the additional ML response 280 from "Advisor" includes a natural language text response to the user input ("Done! That saved 200 grams"). Note that the additional ML response 280 will also include a modified first design object 410, which is automatically included and displayed in the design space 230.

[0077] In some embodiments, for each user input entered into prompt space 220 relating to a selected design object 144 or an empty space within design space 230, intent manager 240 automatically retrieves additional context information 268 and includes the additional context information 268 in prompt 260 generated for that user input. For example, additional context information 268 may include a snapshot image of the design object 144 captured for the user input, and / or geometry of the selected design object 144. The inclusion of additional context information 268 in prompt 260 to provide context information for the user input advantageously allows ML models 180, 190 to provide a more accurate and useful ML model response 280 to prompt 260.

[0078] In some embodiments, additional context information 268 may be retrieved from context data 146 stored in local data storage 140. Context data 146 may include data generated and stored by the design exploration application 130 during the execution of the design application to design and display the design space 230. For example, context data 146 may include recently executed design application commands (such as the last 10 most recent commands executed within the design space 230), or recently executed design application commands associated with the selected design object 144 (such as the last 10 most recent commands executed on the selected design object 144). Context data 146 may further include, but is not limited to, the current state of the selected design object 144 and / or the design space 230, the geometry of one or more other design objects 144 connected to the selected design object 144, the current material associated with the selected design object 144, and so on. Context data 146 may further include, but is not limited to, the context of the user who created the design object 144, such as the design tools typically used, how long the user has used the design application, their level of expertise, etc.

[0079] In some embodiments, additional contextual information 268 may be retrieved from one or more remote contextual databases 194. For example, the remote contextual database 194 may include a database of manufacturers of assemblies and / or parts, which includes specification information for the assemblies and / or parts. In these embodiments, the additional contextual information 268 may include manufacturer specifications for selected design objects 144 (such as selected assembly objects or part objects), such as manufacturer name, assembly / part name, model type, common or optional / possible physical dimensions (width, height, depth), common or optional / possible weight or mass, common or possible materials, common or optional / possible manufacturing processes used, environmental information (such as implicit carbon or implicit greenhouse gas emissions released during the life cycle), etc.

[0080] Display prompt history within the design space Figures 9-13 A cue history feature is shown, which allows a user to view the cue history associated with a specific design object 144 or empty location within design space 230. In these embodiments, design exploration application 130 displays zero or more cue history markers 222 within design space 230. Each cue history marker 222 represents a specific cue history table 252 and cue history associated with a specific design object 144 or empty location within design space 230. Design exploration application 130 may display cue history markers 222 that are close to / adjacent to the corresponding design object 144 or empty location within design space 230. The user can then select a cue history marker 222, and in response, design exploration application 130 retrieves the corresponding cue history table 252 and displays the corresponding cue history associated with the specific design object 144 or empty location. In some embodiments, various innovative functions and processes for implementing the cue history feature are performed by design exploration application 130 and various modules of design exploration application 130, such as Figure 2 As shown.

[0081] Figure 9 According to various embodiments Figure 4 An exemplary illustration of the history markers displayed within the design space. (See reference) Figure 8Upon receiving the additional ML response 280, the user wishes to exit / disable the interaction mode of the cue space 520 and close the cue space 520. For example, the interaction mode of the cue space 520 can be disabled and the cue space 520 closed by selecting the "X" icon in the upper corner of the cue space 520, entering a specific hotkey (a predetermined hotkey combination), or entering a predetermined voice command. In response to the interaction mode being disabled, the design exploration application 130 closes the cue space 520 and displays a first cue history marker 922 near the first design object 410 within the design space 230. Generally, a technique similar to that described above, placing the cue space 220 near the currently selected location or the currently selected design object 144 within the design space 230, can be used to place the cue history marker 922 near the currently selected location or the currently selected design object 144 within the design space 230.

[0082] The first prompt history marker 922 indicates the existence of a first prompt history associated with the first design object 410 that the user may wish to review and explore. The first prompt history associated with the first design object 410 is stored in and tracked in a corresponding first prompt history table 252 generated for the first design object 410. The first prompt history marker 922 may also display a count of the total number of prompt interactions included in the first prompt history. Figure 9 In the example, the first cue history marker 922 displays a count value of "4" because there are a total of 4 cue interactions included in the first cue history.

[0083] While the user explores and modifies the design space 230, the design exploration application 130 continues to display the first hint history marker 910. Then, at any later time, the user can select the first hint history marker 910 to view the first hint history associated with the first design object 410. In other embodiments, the user can view the first hint history by selecting the first design object 410 and then entering a specific hotkey for accessing the hint space. In response to the user's selection to view the first hint history, the design exploration application 130 retrieves and uses the corresponding first hint history table 252 to display the corresponding first hint history associated with the first design object 410, which in... Figure 8 As shown in the image. Figure 8 As shown, in order to display the first prompt history, the design exploration application 130 can open and display the prompt space 520 placed near the first design object 410, and display the prompt interactions of the first prompt history. Through the prompt space 520, the user can view the first prompt history, and can also enter new user input to generate new prompts 260, which are then added to the first prompt history and the first prompt history table 252 for the first design object 410.

[0084] The design exploration application 130 can display multiple prompt history markers 222 for multiple different design objects 144 or empty locations within the design space 230 that the user has interacted with. For example, if no design object 144 or empty location is currently selected, and the user enters a predetermined hotkey combination that triggers the display of multiple prompt history markers 222 within the design space 230, the design exploration application 130 can display multiple prompt history markers 222 within the design space 230.

[0085] However, if an excessive number of cue history markers 222 are displayed in the design space 230, these markers can distract the user. To reduce this visual clutter, the design exploration application 130 may display the cue history markers 222 based on the current zoom level associated with the design space 230. In these embodiments, the design exploration application 130 provides a zoom tool that allows for different zoom levels for viewing the design space 230. The design exploration application 130 may display specific cue history markers 222 based on the currently selected zoom level, thereby displaying only those cue history markers 222 associated with the design object 144 at the current zoom level, and excluding those cue history markers 222 associated with the design object 144 that are not associated with the current zoom level. In some embodiments, cue history markers 222 associated with empty locations are unaffected by the current zoom level and continue to be displayed regardless of the current zoom level.

[0086] Figure 10 According to various embodiments Figure 2 An exemplary illustration of assembly-level prompt history markers displayed within design space 230. Figure 10 In the example, the current zoom level includes a first low "zoom-out" level corresponding to the assembly level zoom for easy viewing of the assembly object. Therefore, the first zoom level is associated with the assembly object and the tooltip history marker 222 associated with the assembly object. As shown, the design exploration application 130 only displays those tooltip history markers 222 associated with the design object 144 that includes the assembly, and not those associated with the design object 144 that includes parts or components. As shown, the design exploration application 130 displays a first tooltip history marker 1022 associated with the first assembly object 1010 within the design space 230. For example, a user can view the tooltip history associated with the first assembly object 1010 by selecting the first tooltip history marker 1022, where the tooltip history is displayed in the tooltip space 1020, which is open and positioned close to the first assembly object 1010.

[0087] Figure 11 According to various embodiments Figure 10 An exemplary illustration of a set of part-level tooltips and history markers displayed within the design space. Figure 11 In the example, the scaling level has been changed from Figure 10 The zoom level is increased to include a second, medium "neutral zoom" level corresponding to the part-level zoom for easy viewing of part objects. Therefore, the second zoom level is associated with the part object and the tooltip history markers 222 associated with the part object. As shown, the design exploration application 130 only displays those tooltip history markers 222 associated with the design object 144 that includes the part, and not those associated with the design object 144 that includes assemblies or components. As shown, the design exploration application 130 displays a first tooltip history marker 1112 associated with the first part object 1110 and a second tooltip history marker 1122 associated with the second part object 1120 within the design space 230. For example, a user can view the tooltip history associated with the first part object 1110 by selecting the first tooltip history marker 1112, where the tooltip history is displayed in the tooltip space 1130, which is opened and positioned close to the first part object 1110.

[0088] Figure 12 According to various embodiments Figure 10 An exemplary illustration of a set of component-level tooltips and history markers displayed within the design space. Figure 12 In the example, the scaling level has been changed from Figure 11 The zoom level is increased to a third high "zoom" level corresponding to the component-level zoom for easy viewing of component objects. Therefore, the third zoom level is associated with the component object and the hint history markers 222 associated with it. As shown, the design exploration application 130 only displays those hint history markers 222 associated with the design object 144 that includes the component, and not those associated with the design object 144 that includes the assembly or part. As shown, the design exploration application 130 displays a first hint history marker 1212 associated with the first component object 1210 (a surface element of a part) and a second hint history marker 1222 associated with the second component object 1220 (an edge element of a part) within the design space 230. For example, a user can view the hint history associated with the first component object 1210 by selecting the first hint history marker 1212, where the hint history is displayed in the hint space 1230, which is opened and positioned close to the first component object 1210.

[0089] In some embodiments, for each user input entered into the cue space 220 and associated with the design object 144, the design exploration application 130 also captures the user viewpoint of the design object 144 at the point in time the user input was entered into the cue space 220. The captured user viewpoint is then stored in a corresponding cue history table 252, which represents the cue history associated with the design object 144, thereby capturing a viewpoint that is also associated with a specific user input within the cue history. The user viewpoint includes the camera viewpoint of the design object 144 at the time the user input was entered. (As stated above regarding...) Figure 3 The camera viewpoint discussed can be captured by recording the camera's position (represented by xyz coordinates) and orientation (represented by a vector) when user input is entered.

[0090] In some embodiments, when the user later selects a prompt history marker 222 representing a prompt history for a specific design object 144 and a prompt history table 252, the design exploration application 130 opens a prompt space 222 displaying the prompt history for the specific design object 144, which includes one or more user inputs. When the prompt history is initially displayed in the prompt space 222, the design exploration application 130 can automatically scroll to the last user input in the prompt history and recreate the user viewpoint of the specific design object 144 at the time the last user input was entered. The user viewpoint for the last user input can be recreated by applying the camera perspective (camera position and camera orientation) captured for the last user input to the specific design object 144 within the design space 230. In this way, the initial user viewpoint of the specific design object 144 displayed in the design space 230 is changed to the captured user viewpoint of the specific design object 144 at the time the last user input was entered, which advantageously provides context and a better understanding of the last user input. When the user closes the prompt space 220 that displays the prompt history, the design exploration application 130 then changes the user viewpoint of the specific design object 144 back to the initial user viewpoint.

[0091] In other embodiments, the user can select specific user input from the prompt history, and in response, the design exploration application 130 can similarly apply the user viewpoint (camera perspective) captured for the selected user input to the design object 144 within the design space 230. In this way, the initial user viewpoint of the specific design object 144 displayed in the design space 230 is changed to the captured user viewpoint of the specific design object 144 when the selected user input was entered, which advantageously provides context and a better understanding of the selected user input. When the user closes the prompt space 220 displaying the prompt history, the design exploration application 130 then reverts the user viewpoint of the specific design object 144 back to the initial user viewpoint.

[0092] In an alternative embodiment, a user viewpoint for user input on the design object 144 can be captured via a snapshot image of the design object 144. (As stated above regarding...) Figure 3 As discussed, links to snapshot images can be stored in a prompt history table 252 for design object 144 and associated with user input. In these embodiments, when a user views the prompt history for design object 144 displayed in prompt space 220, the user can select a specific user input. In response, design exploration application 130 can use prompt history table 252 to retrieve an image captured for the selected user input and display the image near the selected user input in prompt space 220. In some embodiments, the user selects a specific user input by hovering the cursor over it. In this way, the user can quickly view images of viewpoints associated with multiple user inputs in the prompt history by sequentially hovering the cursor over multiple user inputs.

[0093] Figure 13 According to various embodiments Figure 2 An exemplary illustration of a viewpoint image displayed in a prompt space. As shown, the prompt history for a specific design object 144 is displayed in a prompt space 1320 displayed within the design space 230. The prompt history includes multiple prompt interactions, including user input 1310 selected by the user (e.g., by hovering the cursor over user input 1310). In response to the selection of user input 1310, the design exploration application 130 retrieves and displays a viewpoint image 1330 of the specific design object 144, which was captured when the selected user input 1310 was entered. The viewpoint image 1330 can be retrieved via a link stored in a prompt history table 252 associated with the specific design object 144. As shown, the viewpoint image 1330 is displayed close to the selected user input 1310.

[0094] Figure 14 A flowchart illustrating method steps for integrating a prompting space 220 within a design space 230, according to various embodiments, is provided. (Although references are not included...) Figures 1-13 The system describes the method steps, but those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of the embodiments. In some embodiments, the prompt placement method 1400 is performed by the design exploration application 130 and various modules of the design exploration application 130 to implement the prompt placement feature within the design space 230. In some embodiments, method 1400 is related to... Figure 15 The described prompt history method 1500 is executed in combination, and method 1500 implements the prompt history feature within the design space 230.

[0095] As shown in the figure, method 1400 begins at step 1410, where design exploration application 130 displays a GUI 120 including design space 230. Design space 230 displays one or more design objects 144. In some embodiments, GUI 120 displays only design space 230 and prompt space 220 is not currently displayed. Design exploration application 130 then (in step 1420) receives user selection of a location within design space 230, such as selecting a specific design object 144 or an empty location within design space 230. Design exploration application 130 also receives user invocation of prompt space 220, such as user input of a predetermined hotkey combination or predetermined voice command, which activates the invocation interaction mode of prompt space 220.

[0096] In response to step 1420, the design exploration application 130 then (in step 1430) opens and displays a tooltip space 220 at a placement location within the design space 230, based on the currently selected location within the design space 230, such as the location of the currently selected design object 144 or a currently selected empty location. In some embodiments, the tooltip space 220 is a panel / window integrated within / inside the design space 230, whereby at least a portion of the tooltip space 220 overlaps / covers at least a portion of the design space 230. In some embodiments, the tooltip space 220 is displayed near / adjacent to the currently selected design object 144 or a currently selected empty location within the design space 230.

[0097] The design exploration application 130 then (in step 1440) receives user input (such as text input) within the prompt space 220 and stores the data associated with the user input in the prompt history table 252 corresponding to the currently selected design object 144 or an empty location. If the user input is the first user input received for the currently selected design object 144 or an empty location, the design exploration application 130 first generates the corresponding prompt history table 252 and stores it in memory 116. The design exploration application 130 may further capture a snapshot image of the currently selected design object 144 and store it in memory 116. The data captured for the user input may include text input and viewpoint information (such as camera viewpoint information and / or links to the snapshot image). The design exploration application 130 also increments the prompt interaction count stored in the corresponding prompt history table 252.

[0098] The design exploration application 130 then (in step 1450) generates and sends a prompt 260 in response to user input to the server device 160. The prompt 260 may include user input, design intent text 262, one or more design files 264, one or more design space references 266, and / or context information 268 (such as context data 146 and data retrieved from one or more remote context databases 194). The design exploration application 130 then (in step 1460) receives an ML response 280 from the server device 160 in response to the prompt 260. The ML response 280 includes a text response displayed in the prompt space 220 and stored in a corresponding prompt history table 252. The design exploration application 130 also increments the prompt interaction count stored in the corresponding prompt history table 252. The ML response 280 may also include geometry for a new or modified design object 144 displayed in the design space 230.

[0099] Design exploration application 130 then (in step 1470) determines whether an option to close prompt space 220 (exit the interactive mode of prompt space 220) has been received, such as receiving an selection of the "X" icon in prompt space 220, receiving a predetermined hotkey combination, or receiving a predetermined voice command. If no option to close prompt space 220 is received (in step 1470 – No), method 1400 continues in step 1440 to receive new user input in prompt space 220. If an option to close prompt space 220 is received (in step 1470 – Yes), design exploration application 130 closes prompt space 220 (in step 1480). Method 1400 may then be repeated in step 1410, or continue to step 1510 in method 1500.

[0100] Figure 15 A flowchart illustrating method steps for displaying a prompt history within a prompt space, according to various embodiments, is provided. (Although references are provided...) Figures 1-13 The system describes the method steps, but those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of the embodiments. In some embodiments, the prompt history method 1500 is executed by the design exploration application 130 and various modules of the design exploration application 130 to implement prompt history features within the design space 230. In some embodiments, method 1500 is related to... Figure 14 The described tooltip placement method 1400 is executed in combination, and method 1400 implements the tooltip placement feature within the design space 230.

[0101] As shown in the figure, method 1500 begins at step 1510, where design exploration application 130 displays GUI 120 including design space 230. Design space 230 displays one or more design objects 144 from an initial viewpoint. In some embodiments, GUI 120 displays only design space 230 and tooltip space 220 is not yet displayed.

[0102] The design exploration application 130 also (in step 1520) displays a cue history marker 222 within the design space 230 for each design object 144 or empty location, each design object or empty location having an associated cue history stored in the corresponding cue history table 252. Each cue history marker 222 is displayed near the associated design object 144 or empty location within the design space 230 to indicate the presence of a cue history for the design object 144 or empty location. Each cue history marker 222 may display a count of the total number of cue interactions included in the corresponding cue history. For example, the design exploration application 130 may be activated to display the cue history marker 222 in response to a user entering a predetermined hotkey combination or a specific voice command.

[0103] In some embodiments, to reduce visual clutter, in step 1520, the design exploration application 130 displays only some cue history markers 222 based on the current zoom level associated with the design space 230. This means only those cue history markers 222 associated with the design object 1440 and not those associated with the design object 1440 but not with the current zoom level. For example, the design exploration application 130 may determine that the current zoom level of the design space 230 corresponds to an assembly-level zoom, then identify only those cue history tables 252 associated with the design object 144 that includes the assembly, and display only the cue history markers 222 for those design objects 144. In some embodiments, cue history markers 222 associated with empty locations are unaffected by the current zoom level and continue to be displayed regardless of the current zoom level.

[0104] The design exploration application 130 then (in step 1530) receives a user selection for a specific cue history marker 222 associated with a particular design object 144 or empty location within the design space 230. In response to step 1530, the design exploration application 130 then (in step 1540) opens and displays the cue space 220 near the associated design object 144 or empty location. The design exploration application 130 retrieves the cue history stored in the cue history table 252 corresponding to the associated design object 144 or empty location and displays the cue history in the cue space 220. In some embodiments, only the cue history for the associated design object 144 or empty location is displayed in the cue space 220, thus cue history for other design objects 144 or empty locations is not displayed in the cue space 220. The cue history may include one or more user inputs, including the last user input. In some embodiments, the design exploration application 130 also recreates the user viewpoint associated with the last user input by applying a camera viewpoint recorded for the last user input to the associated design object 144, so that the associated design object 144 is displayed within the design space 230 at the recorded camera viewpoint.

[0105] The design exploration application 130 then (in step 1550) receives a user selection of a specific user input within the prompt history displayed in the prompt space 220. For example, the user selection may include clicking the specific user input or hovering the cursor over the specific user input. In response to step 1550, the design exploration application 130 then (in step 1560) applies a viewpoint associated with the selected user input. For example, if the user selection includes clicking the user input, the design exploration application 130 may apply a camera view recorded for the selected user input to the associated design object 144 so that the associated design object 144 is displayed within the design space 230 in the recorded camera view. For example, if the user selection includes hovering the cursor over the user input, the design exploration application 130 may display a snapshot image captured for the associated design object 144 when the user input was entered. The snapshot image may be displayed within the prompt space 220 near the selected user input.

[0106] Design exploration application 130 then (in step 1570) determines whether an option to close prompt space 220 has been received, such as receiving an selection of the “X” icon in prompt space 220, receiving a predetermined hotkey combination, or receiving a predetermined voice command. If no selection to close prompt space 220 has been received (in step 1570 – No), method 1500 continues in step 1550 to receive a selection for another user input in prompt space 220. If an selection to close prompt space 220 has been received (in step 1570 – Yes), design exploration application 130 (in step 1580) closes prompt space 220 and returns to the initial viewpoint of design space 230. Method 1500 may then be repeated in step 1510, or continue to step 1410 in method 1400.

[0107] System Implementation Method Figure 16 An architecture for a system in which various embodiments can be implemented is described. In some embodiments, Figure 1 The client device 110 and server device 160 may each be implemented as the system 1600 described herein. The accompanying drawings are in no way limited or intended to limit the scope of this disclosure. In various specific embodiments, system 1600 may be an augmented reality, virtual reality, or mixed reality system or device, a personal computer, a video game console, a personal digital assistant, a mobile phone, a mobile device, or any other device suitable for practicing one or more embodiments of this disclosure. Furthermore, in various embodiments, any combination of two or more systems 1600 may be coupled together to practice one or more aspects of this disclosure.

[0108] As shown in the figure, system 1600 includes a central processing unit (CPU) 1602 and system memory 1604 that communicate via a bus path, which may include a memory bridge 1605. CPU 1602 includes one or more processing cores and, in operation, is the main processor of system 1600 that controls and coordinates the operation of other system components. System memory 1604 stores software applications and data used by CPU 1602. CPU 1602 runs software applications and, optionally, an operating system. For example The memory bridge 1605 of the Northbridge chip is connected via a bus or other communication path ( For example The hyperlink is connected to the I / O (input / output) bridge 1607. It can be... For example The southbridge chip's I / O bridge 1607 connects to one or more user input devices 1608. For exampleThe system receives user input via a keyboard, mouse, joystick, digitizer, touchpad, touchscreen, still or video camera, motion sensor, and / or microphone, and forwards the input to the CPU 1602 via memory bridge 1605.

[0109] The display processor 1612 is connected via a bus or other communication path ( For example The display processor 1612 is coupled to the memory bridge 1605 via a PCI Express, accelerated graphics port, or hyperlink; in one embodiment, the display processor 1612 is a graphics subsystem including at least one graphics processing unit (GPU) and graphics memory. The graphics memory includes display memory for storing pixel data for each pixel of the output image. For example (Frame buffer). The graphics memory can be integrated into the same device as the GPU, connected to the GPU as a separate device, and / or implemented within system memory 1604.

[0110] The display processor 1612 periodically delivers pixels to the display device 1610. For example (Screen or conventional CRT, plasma, OLED, SED, or LCD-based monitors or televisions). Additionally, the display processor 1612 can output pixels to a film recorder suitable for reproducing computer-generated images on photographic film. The display processor 1612 can provide analog or digital signals to the display device 1610. In various embodiments, one or more of the various graphical user interfaces described in Appendix AJ appended herein are displayed to one or more users via the display device 1610, and one or more users can input data to and receive visual output from those various graphical user interfaces.

[0111] System disk 1614 is also connected to I / O bridge 1607 and can be configured to store content, applications, and data for use by CPU 1602 and display processor 1612. System disk 1614 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROMs, DVD-ROMs, Blu-ray, HD-DVDs, or other magnetic storage devices, optical storage devices, or solid-state storage devices.

[0112] The switch 1616 provides I / O bridge 1607 with connectivity to other components such as network adapter 1618 and various plug-in cards 1620 and 1621. Network adapter 1618 allows system 1600 to communicate with other systems via electronic communication networks and may include wired or wireless communication via local area networks and wide area networks (such as the Internet).

[0113] Other components (not shown), including USB or other port connections, film recording devices, etc., may also be connected to I / O bridge 1607. For example, an audio processor can be used to generate analog or digital audio output from instructions and / or data provided by CPU 1602, system memory 1604, or system disk 1614. Figure 1 The communication paths for interconnecting the various components can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express (PCI-E), AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol, and connections between different devices can use different protocols, as known in the art.

[0114] In one embodiment, the display processor 1612 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry, and constitutes a graphics processing unit (GPU). In another embodiment, the display processor 1612 incorporates circuitry optimized for general-purpose processing. In yet another embodiment, the display processor 1612 may be integrated with one or more other system elements, such as memory bridge 1605, CPU 1602, and I / O bridge 1607, to form a system-on-a-chip (SoC). In a further embodiment, the display processor 1612 is omitted, and the functions of the display processor 1612 are performed by software executed by CPU 1602.

[0115] Pixel data can be provided directly from CPU 1602 to display processor 1612. In some embodiments of this disclosure, instructions and / or data representing a scene are provided to a rendering pool or a set of server computers, each similar to system 1600, via network adapter 1618 or system disk 1614. The rendering pool uses the provided instructions and / or data to generate one or more rendered images of the scene. These rendered images can be stored in a digital format on a computer-readable medium and optionally returned to system 1600 for display. Similarly, stereoscopic image pairs processed by display processor 1612 can be output to other systems for display, stored on system disk 1614, or stored in a digital format on a computer-readable medium.

[0116] Alternatively, CPU 1602 provides display processor 1612 with data and / or instructions defining the desired output image, and display processor 1612 generates pixel data for one or more output images based on this data and / or instructions, including characterizing and / or adjusting offsets between stereoscopic image pairs. The data and / or instructions defining the desired output image may be stored in system memory 1604 or graphics memory within display processor 1612. In embodiments, display processor 1612 includes 3D rendering capabilities for generating pixel data for the output image based on instructions and data defining the geometry, lighting and shadows, texturing, motion, and / or camera parameters of a scene. Display processor 1612 may further include one or more programmable execution units capable of executing shader programs, tone mapping programs, etc.

[0117] Furthermore, in other embodiments, the CPU 1602 or display processor 1612 may be replaced or supplemented by any technically feasible form of processing device configured to process data and execute program code. Such processing device may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. In various embodiments, any of the operations and / or functions described herein may be performed by the CPU 1602, the display processor 1612, or one or more other processing devices or any combination of these different processors.

[0118] CPU 1602, render field and / or display processor 1612 may employ any surface or volume rendering technique known in the art to create one or more rendered images based on provided data and instructions, including rasterization, scanline rendering, REYES or micropolygon rendering, ray casting, ray tracing, image-based rendering techniques and / or combinations of these techniques and any other rendering or image processing techniques known in the art.

[0119] In other envisioned embodiments, system 1600 may be a robot or robotic device and may include CPU 1602 and / or other processing units or devices, as well as system memory 1604. In such embodiments, system 1600 may or may not include... Figure 1 Other elements shown. System memory 1604 and / or other memory cells or devices in system 1600 may include instructions that, when executed, cause the robot or robot device represented by system 1600 to perform one or more operations, steps, tasks, etc.

[0120] It should be understood that the system shown herein is illustrative, and variations and modifications are possible. The connection topology, including the number and arrangement of bridges, can be modified as needed. For example, in some embodiments, system memory 1604 is connected directly to CPU 1602 instead of via a bridge, and other devices communicate with system memory 1604 via memory bridge 1605 and CPU 1602. In other alternative topologies, processor 1612 is shown connected to I / O bridge 1607 or directly to CPU 1602, instead of to memory bridge 1605. In other embodiments, I / O bridge 1607 and memory bridge 1605 may be integrated into a single chip. Specific components shown herein are optional; for example, any number of cards or peripherals may be supported. In some embodiments, switch 1616 is eliminated, and network adapter 1618 and cards 1620, 1621 are directly connected to I / O bridge 1607.

[0121] In summary, the disclosed technology can be used to generate 2D or 3D designs in a design space based on the design intent expressed by the user via a cue space in the GUI of the design exploration application 130. The cue space enables the user to generate cue points for input into one or more ML models and receive ML responses to those cue points. The design exploration application 130 initially displays the design space 230 without displaying the cue space 220, allowing the user to explore and navigate the design space 230 without visual clutter. For example, the user can move the cursor to select various design objects 144 and / or empty locations without design objects 144 within the design space 230. If the user expects to generate cue points 260 to receive design guidance or assistance from ML models 180, 190, the user can invoke the cue space 220 anywhere within the design space 230. In response to the user's invocation of the cue space 220, the design exploration application 130 can spatially place / position the cue space 220 within the design space 230.

[0122] In some embodiments, the tooltip space 220 is a panel / window integrated within / inside the design space 230, whereby at least a portion of the tooltip space 220 overlaps / covers at least a portion of the design space 230. In some embodiments, the design exploration application 130 displays the tooltip space 220 at a location within the design space 230 based on the current position of the user-controlled cursor. For example, the design exploration application 130 can display the tooltip space 220 at a location within the design space 230 based on the position of the currently selected design object 144 by placing the tooltip space 220 near / adjacent to the currently selected design object 144 within the design space 230. For example, the design exploration application 130 can display the tooltip space 220 at a location within the design space 230 based on the currently selected empty location (where no design object 144) within the design space 230 by placing the tooltip space 220 near / adjacent to the currently selected empty location within the design space 230.

[0123] The prompt space 220 is used to interact with ML models 180, 190 by submitting a prompt 260 and then displaying an ML model response 280 to the prompt 260. The prompt space 220 can be used to perform one or more iterations of: (1) entering user input / submitting prompt 260 and (2) receiving ML response 280. After one or more iterations of (1) entering user input / submitting prompt 260 and (2) receiving ML response 280, the prompt space 220 effectively displays a prompt history, which includes all prompt interactions received within the prompt space 220 for the selected design object 144 or the selected empty location. The design exploration application 130 stores and maintains one or more prompt histories associated with one or more design objects 144 and / or one or more empty locations within the design space 230 via one or more prompt history tables 252. For the multiple prompt history tables 252, each prompt history table 252 is a separate and independent table associated with a different design object 144 or a different empty location within the design space 230. In this way, the design exploration application 130 can separately store and track multiple different cue histories for multiple different design objects 144 and / or empty locations within the design space 230.

[0124] In some embodiments, the design exploration application 130 displays zero or more cue history markers 222 within the design space 230. Each cue history marker 222 represents a specific cue history table 252 and cue history associated with a specific design object 144 or empty location within the design space 230. The cue history marker 222 may also display a count value for the total number of cue interactions included in the corresponding cue history. The design exploration application 130 may display a cue history marker 222 near / adjacent to the corresponding design object 144 or empty location within the design space 230. The user can then select a cue history marker 222, and in response, the design exploration application 130 retrieves the corresponding cue history table 252 and displays the corresponding cue history associated with the specific design object 144 or empty location. To display the cue history, the design exploration application 130 may open and display a cue space 220 placed near the associated specific design object 144 or empty location.

[0125] In some embodiments, the design exploration application 130 displays specific cue history markers 222 based on the current zoom level selected for viewing the design space 230. In some embodiments, for each user input entered into the cue space 220 and associated with a design object 144, the design exploration application 130 also captures the viewpoint of the design object 144. Later, when the user selects user input in the cue history associated with the design object 144, the design exploration application 130 may apply the captured viewpoint to the design object 144 within the design space 230. In some embodiments, for each user input entered into the cue space 220 and associated with a specific design object 144 or an empty location within the design space 230, the intent manager 240 also retrieves additional context information 268 and includes the additional context information 268 in a cue 260 generated for that user input.

[0126] At least one technical advantage of the disclosed technology over existing technologies is that it integrates the cue space used for interacting with AI models into the design space of a CAD application, which reduces the amount of user experience distraction when performing design operations within the design space. In this respect, the disclosed technology enables the cue space to be displayed near the currently selected location within the design space, such as the location associated with the currently selected 3D object or an empty location. Spatially placing the cue space within the design space in relation to the user's current focus reduces distraction from performing design work within the design space and promotes a more efficient design process compared to conventional implementations where the cue space is separated from and located outside the design space. These technical advantages provide one or more technical advancements superior to existing methods.

[0127] At least one technical advantage of the disclosed technology over the prior art is that it records separate and distinct cue histories for separate and distinct 3D objects and empty locations within the design space. In this regard, in response to a user request to view the cue history associated with a selected 3D object or empty location within the design space, the cue history is automatically displayed near the selected 3D object or empty location within the design space. Therefore, by using the disclosed technology, displaying the cue history associated with only the selected 3D object or empty location allows users to more easily review and understand previous design processes and interactions with the AI ​​model regarding cue history for the selected 3D object or empty location compared to conventional implementations. These technical advantages provide one or more technical advancements superior to existing methods.

[0128] The various aspects of the subject matter described herein are listed in the following numbered clauses.

[0129] 1. In some embodiments, a computer-implemented method for displaying a prompt space includes displaying a design space comprising one or more design objects; receiving a selection of a current position within the design space; and displaying the prompt space at a placement location within the design space based on the current position.

[0130] 2. The computer-implemented method as described in Clause 1, further comprising receiving first user input via the prompt space.

[0131] 3. The computer-implemented method as described in Clause 1 or 2, further comprising generating a first prompt based on the first user input; and sending the first prompt to a trained machine learning (ML) model for processing.

[0132] 4. The computer-implemented method as described in any one of clauses 1-3, further comprising receiving a first ML response to the first prompt from the trained ML model; and displaying the first ML response within the prompt space.

[0133] 5. A computer-implemented method as described in any one of clauses 1-4, wherein the first user input is associated with a first design object within the design space, and further includes: generating a first cue based on the first user input and context information associated with the first design object; and sending the first cue to a trained machine learning (ML) model for processing.

[0134] 6. A computer-implemented method as described in any one of Clauses 1-5, wherein the context information includes data generated by a design application, the design application also generating the design space, wherein the data includes at least one of: a set of executed design commands, the state of the first design object, the state of the design space, the geometry of at least one other design object coupled to the first design object, or the current material associated with the first design object.

[0135] 7. A computer-implemented method as described in any one of clauses 1-6, wherein receiving the selection of the current location includes receiving the selection of a first design object within the design space.

[0136] 8. A computer-implemented method as described in any one of clauses 1-7, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space close to the first design object.

[0137] 9. A computer-implemented method as described in any one of clauses 1-8, wherein receiving the selection of the current location includes receiving the selection of an empty location within the design space.

[0138] 10. A computer-implemented method as described in any one of clauses 1-9, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space near the empty location.

[0139] 11. In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, cause the one or more processors to display a prompt space by performing the following steps: displaying a design space comprising one or more design objects; receiving a selection of a current location within the design space; and displaying the prompt space at a placement location within the design space based on the current location.

[0140] 12. One or more non-transitory computer-readable media as described in Clause 11, further comprising receiving first user input via the prompt space.

[0141] 13. One or more non-transitory computer-readable media as described in Clause 11 or 12, further comprising generating a first prompt based on the first user input; and sending the first prompt to a trained machine learning (ML) model for processing.

[0142] 14. One or more non-transitory computer-readable media as described in any one of clauses 11-13, further comprising receiving a first ML response to the first prompt from the trained ML model; and displaying the first ML response within the prompt space.

[0143] 15. One or more non-transitory computer-readable media as described in any one of clauses 11-14, wherein the first user input is associated with a first design object within the design space, and further includes: generating a first cue based on the first user input and context information associated with the first design object; and sending the first cue to a trained machine learning (ML) model for processing.

[0144] 16. One or more non-transitory computer-readable media as described in any of Clauses 11-15, wherein the context information includes specification information retrieved from a manufacturer's database for the first design object.

[0145] 17. One or more non-transitory computer-readable media as described in any of Clauses 11-16, wherein the contextual information relating to the first design object includes at least one of the model type, possible physical dimensions, possible weight, possible material, or possible manufacturing process associated with the first design object.

[0146] 18. One or more non-transitory computer-readable media as described in any of clauses 11-17, wherein receiving the selection of the current location includes receiving the selection of a first design object within the design space, the first design object comprising an assembly, part, or element.

[0147] 19. One or more non-transitory computer-readable media as described in any one of clauses 11-18, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space close to the first design object.

[0148] 20. In some embodiments, a system includes one or more memories storing instructions; and one or more processors coupled to the one or more memories, the one or more processors performing the following steps when executing the instructions: displaying a design space including one or more design objects; receiving a selection of a current position within the design space; and displaying the prompt space at a placement location within the design space based on the current position.

[0149] Any and all combinations of any element of any claim and / or any element described in this application, in any manner, fall within the scope of this disclosure and protection.

[0150] Various embodiments have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0151] Aspects of embodiments of the present invention may be embodied as systems, methods, or computer program products. Therefore, aspects of this disclosure may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which are generally referred to herein as “modules” or “systems.” Furthermore, any hardware and / or software techniques, processes, functions, components, engines, modules, or systems described in this disclosure may be implemented as circuits or sets of circuits. Additionally, aspects of this disclosure may take the form of computer program products embodied in one or more computer-readable media on which computer-readable program code is embodied. Software construction and entities ( For example Engines, modules, GUIs, etc.) are stored in one or more memories shown in the relevant system diagrams in various embodiments and executed by the processors shown in these same system diagrams.

[0152] Any combination of one or more non-transitory computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination of the foregoing. More specific examples (not an exhaustive list) of computer-readable storage media will include: electrical connections having one or more wires, portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store programs for use with or in connection with an instruction execution system, device, or apparatus.

[0153] The foregoing description of aspects of this disclosure is based on flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks of the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When executed via a processor of a computer or other programmable data processing apparatus, the instructions cause the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams to be implemented. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the blocks may not appear in the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or operation.

[0155] While the foregoing describes embodiments of this disclosure, other and further embodiments of this disclosure may be devised without departing from the basic scope of this disclosure, the scope of which is defined by the appended claims.

Claims

1. A computer-implemented method for displaying a prompt space, the method comprising: Displays a design space containing one or more design objects; Receive the selection of the current location within the design space; as well as Based on the current location, the prompt space is displayed at the designated placement location within the design space.

2. The computer-implemented method of claim 1, further comprising receiving first user input via the prompt space.

3. The computer-implemented method as described in claim 2, further comprising: A first prompt is generated based on the first user input; as well as The first prompt is sent to a trained machine learning (ML) model for processing.

4. The computer-implemented method as described in claim 3, further comprising: Receive a first ML response to the first prompt from the trained ML model; as well as The first ML response is displayed within the prompt space.

5. The computer-implemented method of claim 2, wherein the first user input is associated with a first design object within the design space, and further includes: A first prompt is generated based on the first user input and the context information associated with the first design object; as well as The first prompt is sent to a trained machine learning (ML) model for processing.

6. The computer-implemented method of claim 5, wherein the context information includes data generated by a design application, the design application further generating the design space, wherein the data includes at least one of the following: a set of executed design commands, the state of the first design object, the state of the design space, the geometry of at least one other design object coupled to the first design object, or the current material associated with the first design object.

7. The computer-implemented method of claim 1, wherein receiving the selection of the current location includes receiving the selection of a first design object within the design space.

8. The computer-implemented method of claim 7, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space close to the first design object.

9. The computer-implemented method of claim 1, wherein receiving the selection of the current location includes receiving the selection of an empty location within the design space.

10. The computer-implemented method of claim 9, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space near the empty location.

11. One or more non-transitory computer-readable media, including instructions that, when executed by one or more processors, cause the one or more processors to display a prompt space by performing the following steps: Displays a design space containing one or more design objects; Receive the selection of the current location within the design space; as well as Based on the current location, the prompt space is displayed at the designated placement location within the design space.

12. The one or more non-transitory computer-readable media of claim 11, further comprising receiving first user input via the prompt space.

13. The one or more non-transitory computer-readable media of claim 12, further comprising: A first prompt is generated based on the first user input; as well as The first prompt is sent to a trained machine learning (ML) model for processing.

14. The one or more non-transitory computer-readable media of claim 13, further comprising: Receive a first ML response to the first prompt from the trained ML model; as well as The first ML response is displayed within the prompt space.

15. One or more non-transitory computer-readable media as claimed in claim 12, wherein the first user input is associated with a first design object within the design space, and further comprises: A first prompt is generated based on the first user input and the context information associated with the first design object; as well as The first prompt is sent to a trained machine learning (ML) model for processing.

16. The one or more non-transitory computer-readable media of claim 15, wherein the context information includes specification information retrieved from a manufacturer's database for the first design object.

17. One or more non-transitory computer-readable media as claimed in claim 16, wherein the contextual information for the first design object includes at least one of the following associated with the first design object: model type, possible physical dimensions, possible weight, possible material, or possible manufacturing process.

18. One or more non-transitory computer-readable media as claimed in claim 11, wherein receiving the selection of the current location includes receiving the selection of a first design object within the design space, the first design object comprising an assembly, part, or element.

19. One or more non-transitory computer-readable media as claimed in claim 18, wherein displaying the prompt space at the placement location within the design space includes displaying the prompt space close to the first design object.

20. A system comprising: One or more memories, wherein the one or more memories store instructions; as well as One or more processors, coupled to one or more memories, wherein the one or more processors perform the following steps when executing the instructions: Displays a design space containing one or more design objects; Receive the selection of the current location within the design space; as well as Based on the current location, a prompt space is displayed at the placement location within the design space.

Images