A multimodal user interface for interacting with digital model files.

Abstract

A method and system are provided that enables multimodal input for interacting with living digital objects. The system receives live digital objects, including digital artifacts extracted from digital model files, through model representations. The system receives the user's security level and determines the user's access rights and permission to access and modify digital artifacts. The system accesses a multimodal interface configured to receive conversational and spatial inputs, outputs digital artifacts to the multimodal interface based on access rights, receives conversational and spatial inputs from the multimodal interface, and generates modified digital artifacts from the digital artifacts through digital model representations, based on modification permissions and conversational and spatial inputs. Multimodal interfaces include traditional interfaces (GUI / API), conversational interfaces (text / voice), and spatial computing interfaces (VR / AR / MR / gestures).

Description

Technical Field

[0001] Reference to Related Applications If an Application Data Sheet (“ADS”) or PCT Application Form (“Request”) has been filed on the filing date of this application, it is incorporated herein by reference. The ADS or an application based on 35 U.S.C. §§ 119, 120, 121, or 365(c) in a priority claim, as well as all applications such as parents, grandparents, great-grandparents, etc., are incorporated by reference herein, including any priority claims and materials incorporated by reference within the application, provided they are not inconsistent with this document.

[0002] Furthermore, this application is related to the following U.S. patent applications, which are incorporated herein by reference in their entirety as if fully set forth herein. ● PCT Application No. PCT / US24 / 38878 (Case No.) IST - 03.002 PCT), filed on July 19, 2024, with the title “Generative Artificial Intelligence (AI) for Digital Workflows” describes an efficient AI - assisted script generation method that protects the sovereignty of customer data. ● PCT Application No. PCT / US24 / 35885 (Case No.) IST - 02.002 PCT), filed on June 27, 2024, with the title “Artificial Intelligence (AI) Assists in Integrating New Digital Model Types and Tools into an Integrated Digital Model Platform” describes the enhancement of model splicer technology with AI assistance. ● PCT Application No. PCT / US24 / 27912 (Case No.) IST - 02.003 PCT), filed on May 5, 2024, with the title “Secure and Expandable Sharing of Digital Engineering Documents” represents a secure and scalable document splicing technology. ●PCT application number PCT / US24 / 27898 (case number IST-03.001PCT) was filed on May 4, 2024, and is titled "Digital Twin Enhancement with External Feedback in an Integrated Digital Model Platform Describes the Integration of Digital Twin and Physical Twin Management and External Feedback within the DE Platform." ●PCT application number PCT / US24 / 19297 (case number), IST-01.002PCT, filed on March 10, 2024, with the title "Software Code Definition Digital Thread in an Artificial Intelligence (AI) Assisted Digital Engineering System," describes an AI-assisted digital thread for a digital engineering platform. ●PCT application number PCT / US24 / 18278 (Case number IST-02.001PCT filed on March 3, 2024, title "Secure and Extensible Model Splicing of Digital Engineering Models for Software Code Definition Digital Threads") describes model splicing for digital engineering platforms. ●PCT application number PCT / US24 / 14030 (case number IST-01.001PCT), filed on February 1, 2024, titled "Artificial Intelligence (AI)-Assisted Digital Documentation for Digital Engineering," describes AI-assisted documentation for a digital engineering platform. ● U.S. Provisional Patent Application No. 63 / 442,659 (Case No. IST-01.001P), filed on February 1, 2023, titled "AI-assisted digital documentation and support systems / methods for digital engineering," describes AI-assisted tools for digital engineering (DE), including modeling and simulation applications and certification of digital engineering products. ● U.S. Provisional Patent Application No. 63 / 451,545 (Application No. IST-01.002P), filed on March 10, 2023, is titled "Describing Digital Threads in Digital Engineering Systems, AI-Assisted Digital Thread Generation Support, Model Splicers, and Digital Threading Techniques." ● U.S. Provisional Patent Application No. 63 / 451,577 (Case No.: IST-02.001P1), filed on March 11, 2023, is titled "Model Splicers and Microservice Architectures for Digital Engineering" and describes model splicer technology. ● U.S. Provisional Patent Application No. 63 / 462,988 (Case No. IST-02.001P2), filed on April 29, 2023, with the same title, “Model Splicers and Microservice Architectures for Digital Engineering,” describing model splicer technology. ● U.S. Provisional Patent Application No. 63 / 511,583 (Case No. IST-02.002P), filed on June 30, 2023, is titled "Model Splicer Generation for AI-Assisted Digital Engineering" and describes an AI-assisted model splicer technology. ● U.S. Provisional Patent Application No. 63 / 516,624 (Case No. IST-02.003P), filed on July 31, 2023, is titled "Document and Model Splicing for Digital Engineering" and describes a document splicer technology. ● U.S. Provisional Patent Application No. 63 / 520,643 (Case No.: IST-02.004P), filed on August 20, 2023, is titled "Automation of Artificial Intelligence (AI) Assisted Testing in Software Environments" and describes AI-assisted software testing. ● U.S. Provisional Patent Application No. 63 / 590,420 (Case No.: IST-02.005P), filed on October 14, 2023, titled "Comment and Collaboration Capabilities in a Digital Engineering Platform," describes collaborative capabilities. ● U.S. Provisional Patent Application No. 63 / 586,384 (Case No. IST-02.006P), filed on September 28, 2023, titled "Artificial Intelligence (AI) Assisted Model Splicing, Unit Testing, and Documentation," describes an AI-assisted method for streamlining model splicing, testing, and documentation. ● U.S. Provisional Patent Application No. 63 / 470,870 (Case No.: IST-03.001P), filed on 3 June 2023, titled "Digital Twin and Physical Twin Management with Integrated External Feedback within a Digital Engineering Platform Describes the Integration of Digital Twin and Physical Twin Management and External Feedback within a DE Platform." ● U.S. Provisional Patent Application No. 63 / 515,071 (IST-03.002P, filed July 21, 2023, titled "Generative Artificial Intelligence (AI) for Digital Engineering") describes an AI-powered process for completing digital engineering tasks within a DE software platform. ● U.S. Provisional Patent Application No. 63 / 517,136 (Case No. IST-03.003P), filed on 2 August 2023, titled "A machine learning engine for improving workflows in digital engineering, describing a machine learning engine for model splicing and DE script generation." ● U.S. Provisional Patent Application No. 63 / 516,891 (Case No. IST-03.004P), filed on August 1, 2023, titled "A Multimodal User Interface for Digital Engineering Describes a Multimodal User Interface for a DE System." ● U.S. Provisional Patent Application No. 63 / 580,384 (Case No.: IST-03.006P), filed on 3 September 2023, titled "A Multimodal Digital Engineering Document Interface for Authentication and Security Review Describes a Multimodal User Interface for Authentication and Security Review." ● U.S. Provisional Patent Application No. 63 / 613,556 (Case No. IST-03.008P), filed on December 21, 2023, titled "Selection and Optimization of Alternative Tools in an Integrated Digital Engineering Platform," describes tool selection and optimization. ● U.S. Provisional Patent Application No. 63 / 584,165 (Case No. IST-03.010P), filed on September 20, 2023, titled "Method and System for Workflow Improvement in Digital Engineering," represents workflow optimization in a DE platform. ● U.S. Provisional Patent Application No. 63 / 590,456 (Case No.: IST-04.001P), filed on 15 October 2023, titled "Data Sovereignty Assurance for Artificial Intelligence (AI) Models," relates to data sovereignty assurance in the training and evaluation of AI models. ● U.S. Provisional Patent Application No. 63 / 606,030 (Case No.: IST-04.001P2), filed on December 4, 2023, with the same title, "Data Sovereignty Assurance for Artificial Intelligence (AI) Models, and Further Details on Data Sovereignty Assurance in the Training and Evaluation of AI Models." ● U.S. Provisional Patent Application No. 63 / 419,051, filed on October 25, 2022, titled "Interconnected Digital Engineering and Certification Ecosystem." ● U.S. Non-Provisional Patent Application No. 17 / 973,142 (Case No. 54332-0057001) was filed on October 25, 2022, and is titled "Interconnected Digital Engineering and Certification Ecosystems." ● U.S. Non-Provisional Patent Application No. 18 / 383,635 (Application No. 54332-0059001), filed October 25, 2023, "Interconnected Digital Engineering and Certification Ecosystem." ● U.S. Provisional Patent Application No. 63 / 489,401, filed on March 9, 2023, titled "Security Architecture for Interconnected Digital Engineering and Authentication Ecosystems."

[0003] Copyright and Trade Dress Notice This patent document contains material that is protected by copyright. This patent document may indicate or describe matters that are or may become trademarks of their respective owners. The copyright and trade dress owners have no objection to facsimile copies of the patent disclosures found in the files and records of the United States Patent and Trademark Office, but otherwise reserve all rights to their copyrights and trade dress.

[0004] ISTARI DIGITAL is a trademark name that includes specific examples of the present invention, and therefore the aforementioned trademark name may be used synonymously in the specifications and drawings to refer to the products / processes provided by the examples of the present invention. In these specifications, the terms ISTARI and ISTARI DIGITAL may be used to refer to the present invention and the company providing the invention.

[0005] Field of Invention This invention relates to a multimodal user interface, conversational and spatial computing interface for digital software platforms.

[0006] Background of the Invention The background description of the invention is intended to help understand the invention and its applications and uses, and is not considered prior art.

[0007] Digital engineering, and more broadly digital model software, is a field poised to see interactive technologies and techniques such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) play a crucial role as interfaces between users and emerging digital engineering platforms. These technologies are being leveraged to create immersive experiences that allow users to interact with digital environments. Other areas of technology adoption include natural language processing (NLP), machine learning (ML), and artificial intelligence (AI). These technologies are being used to create more intelligent and intuitive interfaces that can adapt to the needs of individual users. There are several challenges to integrating these technologies, including the fragmentation of digital engineering applications. Furthermore, the fact that feedback sources and digital models are likely to originate from different clients and organizations can exacerbate security issues associated with iterative digital engineering in some ways.

[0008] Conversational interfaces, both text-based and voice-based, are becoming increasingly popular in a variety of applications. These interfaces, such as Siri and Alexa, have been particularly successful in the field of home automation, providing users with hands-free and intuitive device operation. However, their application in the field of digital modeling platforms is limited. Existing conversational interfaces are primarily designed for general use and lack the specialized capabilities required for digital modeling platforms. These platforms involve complex operations and require precise control that is difficult to achieve with the broad commands commonly used in general voice interfaces. Furthermore, these interfaces often struggle to handle the technical and specialized terminology used in the field of digital modeling. Another limitation of existing conversational interfaces is their inability to handle complex queries and commands. In the context of digital modeling platforms, users need to perform multi-stage operations across multiple tools and manipulate large amounts of data, tasks that current conversational interfaces cannot handle.

[0009] Gesture and spatial interfaces have also been developed for specific applications such as controlling game consoles, televisions, and projectors. These interfaces allow users to interact with digital systems using gestures and movements, providing a more immersive and interactive experience. However, these interfaces often have limitations in their ability to handle precise engineering and model-related tasks that require highly accurate interaction with digital systems.

[0010] Code interfaces such as APIs have been used to interact with digital systems. These interfaces allow users to interact with digital systems using programming languages, providing a high level of control and flexibility. However, these interfaces often require advanced technical expertise, which can be a barrier for non-technical people.

[0011] Therefore, a conversational interface, including both text and voice, specifically designed for digital modeling platforms is needed. Such an interface must handle the complex operations and technical terminology inherent in these platforms and provide a way for users to interact with the tools more intuitively and efficiently. Specifically, such an interface should simplify the user's operation when performing desired actions on specific digital software tools.

[0012] Furthermore, there is a need to integrate multimodal interfaces, including more versatile and advanced interfaces (e.g., both spatial and conversational interfaces, as well as traditional user interfaces), to handle a wide range of software tasks and provide a model collaboration platform accessible to both technical and non-technical users.

[0013] Therefore, integrating interdisciplinary engineering models gathered from different, isolated tools and enabling multimodal interfaces within a unified, scalable, collaborative, and secure digital model platform will represent a cutting-edge advancement.

[0014] Against this backdrop, various practical examples of the present invention have been developed. [Overview of the Initiative]

[0015] This summary of the invention provides a general overview of the invention, its uses, and applications, and is not intended to limit the scope of the invention, which will become clear from the detailed description when read in conjunction with the drawings.

[0016] As described below, and further explained in PCT applications PCT / US24 / 35885 (IST-02.002PCT), PCT / US24 / 27912 (IST-02.003PCT), PCT / US24 / 27898 (IST-03.001PCT), PCT / US24 / 19297 (IST-01.002PCT), PCT / US24 / 18278 (IST-02.001PCT), and PCT / US24 / 14030 (IST-01.001PCT), the emergence of model splicing enables the scripting of digital workflow operations that incorporate different software tools into a corpus of standardized program code. As a result, a large space for digital workflows, including digital model generation, model modification, model data sharing, digital thread generation, thread modification, thread data extraction, thread data sharing, digital twin generation, digital twin modification, digital twin data extraction, digital twin sharing, etc., can be woven into the program code. Next, by translating the operations of the digital workflow into code, it becomes possible to generate and train AI modules intended to manipulate digital models, digital threads, and digital twins.

[0017] The methods and systems described herein use a multimodal interface within a zero-trust environment to enable cross-tool creation, access, and manipulation of digital artifacts, digital model files, digital threads, and digital twins, thereby potentially resulting in a dramatic reduction in cost and latency throughout all stages of the life cycle of any digital design product.

[0018] A live digital object is an interface configured based on the principles of zero-trust security, whether it is a live digital document (live document interface), a live digital board (live dashboard interface), or a live digital space (a 3D virtual or 3D extended environment providing the background context for digital threads and their related artifacts), and is designed to rationalize the digital workflow process for decision-making and review.

[0019] Live digital objects enable the creation of a secure digital thread using optimal data artifacts tailored to the user's needs. However, the user may be overwhelmed when trying to access the most relevant information due to the large number of available digital artifacts. The integration of VR / AR or spatial computing interfaces can be expected to improve the user experience. However, the integration of VR / AR with interconnected digital model / engineering platforms (IDMP / IDEP) gives rise to additional issues related to navigation, sorting, display, security, and real-time collaboration for digital artifacts, digital models, and digital documents.

[0020] The methods and systems disclosed herein provide integration of VR / AR with IDMP and its live digital object interface to enhance zero-trust collaboration while simultaneously resolving related navigation, search, and sorting issues in real time. By leveraging VR / AR, users can interact with digital artifacts spatially and interactively and intuitively, enabling faster navigation and more efficient sorting of digital artifacts and models. This integration also ensures that data is displayed securely while dynamically adhering to user permissions, thus supporting a secure and collaborative environment.

[0021] Compared to conventional VR / AR interfaces for displaying browsers and / or CAD models, the methods and systems disclosed herein extend spatial and interactive interface capabilities to the display and manipulation of common digital artifacts in a zero-trust manner (using AR / VR peripherals, spatial gesture tracking with or without gloves, eye tracking, spatial audio, and / or biometric authentication).

[0022] Accordingly, a process, a system for interacting with live digital objects, and a non-temporary storage medium for storing program code for executing the process are provided. Generally, the present invention relates to methods and systems for digital workflows and digital engineering, and more particularly to multimodal user interfaces for digital workflows and digital engineering systems, including interactive and spatial computing interfaces for human users, as well as code interfaces for bots or human users.

[0023] According to a first aspect, or in one embodiment, a method for performing operations on a computer to interact with a live digital object is disclosed. The method may include receiving a live digital object. The live digital object may include a digital artifact extracted from a digital model file via a model representation. The model representation may include a model-type specific locator for the digital model data. The method may include initiating a connection to a multimodal interface. The multimodal interface may be configured to receive input from at least two different modalities. The multimodal interface may be configured to output at least two different modalities. The at least two different modalities may include at least an interactive modality and a spatial modality. The method may also include receiving the security level of a first user. Based on the security level of the first user, the method may include determining the first user's access rights to access the digital artifact and the first user's modification rights to modify the digital artifact. Based on the first user's access rights to access the digital artifact, the method may include outputting the digital artifact to the multimodal interface via the connection. The method may include receiving interactive and spatial inputs from a first user related to a digital artifact via a multimodal interface and connection. Finally, the method may include generating a modified digital artifact from the digital artifact via a model representation, based on the first user's modification authority to modify the digital artifact and based on at least one of the interactive and spatial inputs.

[0024] In a second aspect or yet another embodiment, a system is provided for interacting with a live digital object, the system comprising at least one processor and at least one memory for storing program code, the program code being executable by at least one processor to cause at least one processor to execute a process for interacting with a live digital object, the program code comprising code for performing the steps described above.

[0025] In a third aspect or yet another embodiment, one or more non-temporary computer-readable storage media are provided for storing program code, the program code being executable by a processor, and when executed by the processor, the program code causes the processor to perform a computerization process for interacting with live digital objects, the program code including code for performing the aforementioned steps.

[0026] In one embodiment, the program code may further include code for displaying a plurality of permitted artifacts based on the security level of a first user. The program code may also include code for receiving spatial input from the first user, which includes a first gesture input for selecting one of the plurality of permitted artifacts. The program code may also include code for receiving spatial input from the first user, which includes a second gesture input for positioning one of the plurality of permitted artifacts. The program code may also include code for updating a live digital object based on the positioning of one of the plurality of permitted artifacts.

[0027] In another embodiment, the program code may further include code for generating an orchestration script that accesses one of several permitted artifacts. The orchestration script may include instructions for generating a new live digital object. The new security level of the new live digital object may be based on a given security level of one of the several permitted artifacts.

[0028] In yet another embodiment, the digital model file may be selected from a group consisting of digital engineering (DE) model files, medical model files, supply chain logistics model files, manufacturing model files, and financial model files.

[0029] In one embodiment, the model representation may include a model splice connected to a digital model file. The model splice may include one or more splice data items, one or more splice data structures, and a splice function that provides access to digital artifacts. Access to digital artifacts may be provided by items selected from a group consisting of application programming interface (API) endpoints and software development kit (SDK) endpoints.

[0030] In another embodiment, live digital objects may be maintained using a software-defined digital thread. The software-defined digital thread may include instructions for accessing model splices. The software-defined digital thread may include instructions for accessing digital model files and different digital model files.

[0031] In yet another embodiment, the digital model file and the different digital model files may come from different software tools. The software code-defining digital thread may access different digital artifacts through different model splices connected to the different digital model files.

[0032] In one embodiment, the program code may further include code for sharing a model splice associated with a modified digital artifact.

[0033] In another embodiment, the program code may further include code for modifying the model representation. The code for modifying the model representation may include code for updating the model splice. The code for updating the model splice may include code for updating the splice function of a model splice selected from a group of splice functions consisting of a given splice data item of the model splice, a given splice data structure of the model splice, and a given splice function of the model splice.

[0034] In yet another embodiment, the modified digital artifact may appear on the live digital object within a predetermined delay.

[0035] In one embodiment, a live digital object may be a live digital document. The live digital document may include one or more digital artifacts obtained from one or more model files within a predetermined delay.

[0036] In another embodiment, the live digital object may be a live digital board. The live digital board can display one or more documents and one or more applications on a 2D screen rendered in a multimodal interface modality selected from the group consisting of a 2D display, a 2.5D display, and a 3D semi-immersive or fully immersive display.

[0037] In another embodiment, a live digital object may be a live digital space. A live digital space can display one or more documents and one or more other applications in a virtual space within a 3D spatial display.

[0038] In one embodiment, live digital objects can be stored and accessed from an interconnected digital model platform (IDMP).

[0039] In another embodiment, the program code may further include code for sending a modified digital artifact to a second user. The program code may further include code for receiving feedback instructions from the second user. In response to an action or approval by the first user, the program code may further include code for modifying a digital model file based on feedback instructions from the second user.

[0040] In yet another embodiment, the live digital object may further include a second digital artifact accessed via a second model representation. The program code may further include code for receiving a second user input from a second user regarding the live digital object. The second user input may be an input in a modality selected from the group consisting of interactive modalities and spatial modalities. The program code may further include code for further modifying the second digital artifact based on the second user input to generate a second modified digital artifact.

[0041] In one embodiment, the live digital object may be an object in an interconnected digital model platform (IDMP). The IDMP may include a natural language processing (NLP) module configured to interact with a first user based on one or more user inputs. The multimodal interface may be further configured to communicate with the first user in natural language.

[0042] In another embodiment, the multimodal interface may include an interactive interface configured to receive audio-based input. One or more user inputs may include voice-based input.

[0043] In yet another embodiment, the multimodal interface may include an interactive interface configured to receive text-based input. At least one of the one or more user inputs may include text-based input.

[0044] In one embodiment, the spatial input may include at least one of augmented reality (AR) input, virtual reality (VR) input, mixed reality (MR) input, and gesture input.

[0045] In another embodiment, the multimodal interface may include a spatial computing interface comprising a video input module, an audio input module, and a gesture input module. Video-based input may be obtained via video received by the video input module. Audio-based input may be obtained via audio received by the audio input module. At least one gesture may be extracted via the gesture input module.

[0046] In yet another embodiment, the spatial computing interface may be further configured to receive contextual information associated with a video-based input.

[0047] In a fourth aspect or yet another embodiment, a non-temporary computer-readable storage medium is provided which stores executable instructions, which, when executed by one or more processors, cause the processors to perform a process for interacting with a live digital object, including the steps described above.

[0048] In a fifth aspect or yet another embodiment, a computer program product is provided. The computer program may be used to interact with a live digital object and may include a computer-readable storage medium in which program instructions or program code are embodied, the program instructions being executable by a processor to cause the processor to perform the steps described above.

[0049] In a sixth aspect or yet another embodiment, a system is provided for interacting with a live digital object, the system including a memory for storing computer executable components, and a hardware processor operably coupled to the memory and for executing the computer executable components stored in the memory, the computer executable components may include components operably coupled to the processor for performing the steps described above.

[0050] In a seventh aspect or yet another embodiment, a system is provided for interacting with a live digital object, the system comprising a processor, a multimodal interface, a user device having a first memory, a server having a second memory and a data repository, a communication link between the aforementioned user device and the aforementioned server, and a plurality of computer codes embodied in the aforementioned first memory and the aforementioned second memory of the aforementioned user device and the aforementioned server, the plurality of computer codes, when executed, cause the aforementioned server and the aforementioned user device to perform a process including the steps described herein.

[0051] In an eighth aspect or yet another embodiment, a computerized server is provided which includes at least one processor, memory, and a plurality of computer codes embodied in the memory, the plurality of computer codes, when executed, cause the processor to execute a process including the steps described herein. Other aspects and embodiments of the present invention include methods, processes, and algorithms including the steps described herein, and also include processes and operating modes of the systems and servers described herein.

[0052] In a ninth aspect or yet another embodiment, an edge computerization system is provided which operates on a physical system or physical twin with access to processing, memory, computer code stored in a non-temporary computer-readable storage medium of a physical system or physical twin, and a plurality of sensor data measured by the aforementioned physical system or physical twin, or on a physical system or physical twin that is dedicated to providing such code and data, the computer code causing a processor to perform the steps described above.

[0053] Features described in the context of different aspects and / or embodiments of the present invention may be used together and / or interchangeable wherever possible. Similarly, where features are described in the context of a single embodiment for the sake of brevity, those features may also be provided separately or in any suitable partial combination. Features described in relation to non-temporary physical storage media may have corresponding features that are definable and / or combinatable with respect to digital documentation systems and / or methods and / or systems, and vice versa, as these embodiments specifically envision.

[0054] Further other aspects and embodiments of the present invention will become apparent from the modes for carrying out the invention when read in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0055] The accompanying drawings incorporated and comprising this specification serve to illustrate specific examples of the invention and to explain the principles of the examples disclosed in conjunction with the description. For clarity, brevity, and flexibility, not all elements, parts, and specifications are defined in every drawing. Not all drawings corresponding to specific processes or embodiments of the invention are drawn to scale. Instead, the emphasis is on describing the nature, function, and product of the manufacturing methods and apparatus described herein.

[0056] The examples of this invention are exemplary and not restrictive. They will now be explained through specific examples with reference to the accompanying drawings.

[0057] Interconnected Digital Engineering / Modeling Platform (IDEP / IDMP) Fig. 1 shows an exemplary interconnected digital model platform (IDMP) architecture in accordance with several examples of the present invention.

[0058] The two figures illustrate an exemplary implementation of IDEP as an interconnected digital engineering (DE) and authentication ecosystem, and an exemplary digital authentication product that conforms to several implementations of the present invention.

[0059] Fig. 3. Another exemplary implementation demonstrating the services and features provided by IDEP in accordance with several implementations of the present invention is shown.

[0060] Fig. 4. The following illustrates several examples of the present invention, illustrating possible scenarios for instantiating IDEP in a customer's physical system and IT environment.

[0061] Fig. 5. According to several examples of the present invention, we present exemplary multimodal interface designs for feedback integration.

[0062] The digital engineering platform links digital models to digital threads. Figure 6 is a schematic comparing exemplary digital threads connecting DE models, according to several examples of the present invention.

[0063] Fig. 7. This is a schematic diagram showing an exemplary DE model splicing setup in accordance with several examples of the present invention.

[0064] Fig. 8 This is a schematic diagram illustrating the digital thread of a DE model by model splicing, according to some examples of the present invention.

[0065] Fig. 9 This is a schematic diagram illustrating the connection of digital splicing with model splicing at the splice surface, according to some examples of the present invention, and showing the difference between digital splicing with and without model splicing.

[0066] Fig. 10 In accordance with several examples of the present invention, we present an exemplary directed acyclic graph (DAG) representation of a pipelined DE task related to a digital thread.

[0067] Multimodal user interface Fig. 11 is an exemplary system diagram illustrating the process of interacting with live digital objects, according to several examples of the present invention.

[0068] Fig. 12. An example workflow is provided illustrating how different user interfaces enable specific user operations within a digital engineering platform, in accordance with the embodiments of the present invention.

[0069] Fig. 13. In accordance with one example of the present invention, a generalized AI-assisted design process is demonstrated on a digital engineering platform.

[0070] Fig. 14 A flowchart illustrating the navigation and sorting of data artifacts in a multimodal interface is shown here, following the example presented.

[0071] Fig. 15 In accordance with one embodiment of the present invention, an exemplary generation and execution of orchestration scripts through a voice / conversation interface is demonstrated.

[0072] Fig. 16 In accordance with one example of the present invention, we demonstrate the exemplary generation and execution of an orchestration script through a spatial computation interface.

[0073] Fig. 17 An example flowchart illustrating aspects of the operation of a disclosed system relating to multimodal communication of a code interface is shown in accordance with the embodiment of the present invention.

[0074] Fig. 18 Based on several examples of the present invention, a descriptive flow diagram is shown for exemplary use cases in which a GUI / API interface is used in an AI-assisted requirements validation process.

[0075] Fig. 19 An example workflow illustrating aspects of the operation of a system disclosed in relation to a voice-controlled interface is shown in accordance with the embodiments of the present invention.

[0076] Fig. Following the examples in Disclosure 20, we present another workflow example illustrating aspects of the operation of the disclosure system related to the voice control interface.

[0077] Fig. 21 Based on several examples of the present invention, a descriptive flow diagram of an exemplary AI-assisted requirements validation process using an interactive interface is shown.

[0078] Fig. 22 Following the examples in Disclosure, an example workflow is shown illustrating aspects of the operation of the disclosure system related to the AI-assisted conversation interface.

[0079] Exemplary graphical user interface (GUI) Fig. 23 According to one embodiment of the present invention, a screenshot of an exemplary graphical user interface (GUI) for manipulating digital threads on IDEP is shown.

[0080] Fig. 24 According to one embodiment of the present invention, a screenshot of another exemplary graphical user interface (GUI) used for manipulating digital threads on IDEP is shown.

[0081] Fig. 25 According to one embodiment of the present invention, an exemplary graphical user interface (GUI) for generating or updating live suites and collaboration boards on IDMP is shown.

[0082] Exemplary multimodal user interface techniques and systems Fig. 26 shows the use of a multimodal interface for accessing data through a virtual live board, following the example shown here.

[0083] Fig. 27 A flowchart illustrating the process of interacting with a live digital object is shown here, following the example provided.

[0084] Fig. 28 A flowchart detailing the process of digital engineering using a multimodal interface is shown here, following the example presented.

[0085] Fig. 29 is an exemplary flowchart illustrating a digital engineering process using a conversational interface, in accordance with several examples of the present invention.

[0086] Machine Learning Implementation Architecture for IDEP / IDMP Operations Fig. 30 The operating principle of a neural network is described according to several examples of the present invention.

[0087] Fig. 31 Based on several examples of the present invention, an overview of the training process for the IDEP neural network is shown.

[0088] Fig. 32 is an explanatory flow diagram illustrating different stages and datasets involved in training an IDEP machine learning model, according to several examples of the present invention.

[0089] Hardware and software architecture for IDEP / IDMP operation Fig. 33 In accordance with several examples of the present invention, we provide illustrative circuit diagrams of the server (management computing entity) and client (user computing entity) used for document creation within IDEP. [Modes for carrying out the invention]

[0090] Detailed description of the invention The following description includes many specific details to ensure a thorough understanding of the invention for illustrative purposes. However, it will be clear to a skilled articulator that the invention can be practiced without these specific details. In other cases, structures, apparatus, activities, methods, and processes are illustrated using schematics, use cases, and diagrams so as not to obscure the invention. The following description includes many details for illustrative purposes, but a skilled articulator will understand that many variations and modifications of the proposed details fall within the scope of the invention. Similarly, many features of the invention are described in relation to or together with others, but a skilled articulator will understand that many of these features can be provided independently of others. Therefore, the description of the invention is written without loss of generality and without imposing limitations on the invention.

[0091] overview This invention broadly relates to methods and systems that enable user control and interaction with digital platforms through diverse interfaces. The embodiment of this invention is sent to a digital platform capable of handling multimodal input and output, making it easier for users to interact with digital models. Multimodal input generally refers to interacting with computers and other electronic devices using multiple means of communication, such as text, voice, images, video, and gestures. Multimodal input is utilized to improve the accessibility and usability of electronic devices for people with disabilities or those who prefer different means of communication. It is also possible to improve the accuracy and efficiency of input methods by combining multiple sources of information.

[0092] The multimodal interface includes text-based conversational and voice-activated interactions, spatial computing including video and gestures, and code-based interactions such as application programming interfaces (APIs). This system provides a comprehensive and versatile platform for digital workflows and digital engineering, enhancing user interaction and efficiency.

[0093] This invention describes an integrated digital model platform that leverages a multimodal interface to enable the efficient creation and management of digital workflows. The embodiment of this invention enables collaboration between different digital models and software tools, allowing industry and the creative metaverse to generate, design, manufacture, and operate digital systems, software tools, and digital models. The embodiment of this invention enables the digital scaling of the complex learning curves associated with digital workflows, significantly accelerating innovation while reducing associated costs and environmental impacts. Furthermore, examples of this invention allow AI to receive and benefit from a rich and accurate real-time data source, as described below.

[0094] Based on the diagrams, the embodiment of the present invention will be described in detail. First, the Digital Model Platform (IDMP) and its Digital Engineering Implementation (IDEP) will be described in detail. Next, the digital splicing and threading operations that enable orchestration script generation will be described in detail. Finally, the multimodal methods and systems will also be described in detail.

[0095] term At the end of this book, several descriptive terms used to aid in understanding the present invention are provided; however, these are not intended to be construed as limiting the scope of the invention. Terms may be used in noun, verb, or adjective form within the scope of their definition.

[0096] Interconnected Digital Model Platform (IDMP) Architecture Fig. 1 This shows an exemplary interconnected digital model platform (IDMP) architecture in accordance with several examples of the present invention. IDMP 100 streamlines the process from conception to production of product development using virtual representations and digital twins. 12 products are used to optimize and refine functionality before creating physical prototypes or physical twins 132, and 122 digital twins 132 are used to iteratively update the digital twins 132 until the substantial twins 132 are synchronized to achieve desired performance targets for the product. In the context of digital engineering (DE), IDMP 100 can be identified as an interconnected digital engineering platform (IDEP).

[0097] Specifically, manufacturers of products (e.g., airplanes, spacecraft, exploration rovers, missile systems, automobiles, railway systems, ships, remotely operated underwater vehicles, robots, drones, medical devices, biomedical devices, pharmaceutical compounds, pharmaceuticals, power generation systems, smart grid measurement and management systems, microprocessors, integrated circuits, buildings, bridges, tunnels, chemical plants, oil and gas pipelines, refineries, etc.) may use the IDEP platform to develop new products from 100 to 220. The manufacturer's engineering team may also create or implement a digital twin of the product in a virtual environment, including detailed computer-aided design (CAD) models and finite element analysis (FEA) or computational fluid dynamics (CFD) simulations of component systems such as airframes, wings, engines, propellers, tail fin assemblies, and aerodynamics. A digital twin (122) virtually represents the design and performance characteristics of a product, allowing the team to optimize and refine its features before creating a physical prototype in a physical environment. In one embodiment, the physical twin (132) may be an existing entity, while the digital twin (122) is a digital instance that replicates the individual configuration of the physical twin (132) as it was when it was built or maintained. In this disclosure, for illustrative purposes only, the digital twin (122) and the physical twin (132) are discussed in the context of building a new product, although those familiar with the technology will understand that the realization of a digital twin (122) and the physical twin (132) can be done in any order depending on the specific use case under consideration.

[0098] 122 digital models (e.g., CAD models, FEA models, CFD models) used to create digital twins are displayed in the model aircraft in Figure 1.80. 1. Also appearing in the model aircraft is a neural network (NN) model. 180 can provide machine learning-based predictive modeling and simulation for the DE process. As a DE model, 182 may be spliced ​​into one or more model splices. For example, 172 and 73170 in one splice plane. Separate digital twins: For example, 122 is instantiated from the splice plane, 170 is done via the application plane, and 160 is done via the application plane. Model splice: 172 may be linked to the following model splice: 1162 appears in the application plane via a platform script or application, and 160 is grouped into a digital thread. Multiple digital threads (e.g., 162 and 163) may further collaborate across various stages and phases of the product lifecycle, from concept and design to testing and production. Digital threads enable seamless data exchange and collaboration between departments and stakeholders, ensuring optimized and validated designs.

[0099] Model splicing provides input / output splicing capabilities that enable access to and modification of DE model data. Therefore, DE tasks related to design updates and digital threads can be represented as scripted interconnection / pipeline tasks arranged in a directed acyclic graph (DAG). See Figure 124 for an example of a DE task DAG, which will be discussed in more detail.

[0100] To enhance the design, external sensory data 140 is collected, processed, and integrated into the application plane 160. This process includes linking data from different sources such as physical sensors 134, physical environment sensors 136, and external data streams 180 such as simulation data from model aircraft. API endpoints provide access to digital artifacts from various environments (e.g., physical twin sensors) 134 data) integrated into a splice plane 122, creating a digital twin. Model splices 170 on the splice plane enable autonomous data linking and digital thread generation, ensuring the digital twin 122 accurately represents the real-world performance and characteristics of the product.

[0101] The accuracy of the digital twin verification 122 allows the engineering team to build or materialize a physical twin 132 based on the same twin configuration (i.e., digital design). The physical prototype 132 can be equipped with numerous sensors, such as 14 accelerometers and 34 temperature sensors, to collect real-time performance data. This data can be compared with the digital twin simulation to verify the product's performance and validate the design.

[0102] Processed sensory data144 can be used to estimate parameters that are difficult to measure directly, such as aerodynamic forces and tire contact patch forces. Such processed sensor data provides additional data to the digital twin122, further improving accuracy and reliability. Processed sensory data144 can be generated from physical environment sensors1, including the physical environment36130, and can also be obtained from other external databases142, as described below.

[0103] During development, feedback from customers and market research can be collected to identify opportunities for improving and adjusting the product design. In the Analysis and Control (ACP) aspect, 150 specialists (SMEs) can analyze processed sensory data 144 and feedback from external experts 114 to make informed decisions about necessary design changes. Such analysis 154 may be enhanced or fully enabled by algorithms (i.e., static program code) or artificial intelligence (AI) modules. Links 162 in the digital thread, 134 and 136, processed sensory data 144, and feedback data from experts 114 are generated in the ACP 150, comparing and analyzing sensor and performance data, leading to modifications of model files through the digital thread.

[0104] In particular, sensory data 1 from the physical environment (44,130) and performance data 1 from the virtual environment (26,120) can be supplied to the comparison engine 152. The comparison engine 152 may include tools that allow platform users to compare various design iterations with each other and with design requirements to identify performance deficiencies and trends and run verification and validation (V&V) tools.

[0105] Model splicing is discussed in more detail in Figs.7 Destination 9. Model splicing enables the scripting of any DE operation, including DE model files, on the model plane.180 Each DE model is associated with a different siloed DE tool. By encoding DE models and DE operations in a unified script corpus, IDEP becomes an aggregator that brings together a wide range of DE activities related to a particular product (e.g., aircraft, spacecraft, exploration rovers, missile systems, automobiles, rail systems, marine vehicles, remotely operated underwater vehicles, robots, drones, medical devices, biomedical devices, pharmaceutical compounds, pharmaceuticals, power generation systems, smart grid measurement and management systems, microprocessors, integrated circuits, buildings, bridges, tunnels, chemical plants, oil and gas pipelines, refineries, etc.) and may also involve threading into program code. Therefore, model splicing enables linking and manipulation of all model files (e.g., 182, 184) associated with specific products within the same interconnected DE platform or DE ecosystem.100 As a result, the generation and training of AI modules (e.g., 182), digital threads (e.g., 162), and digital twins (e.g., 122) for the purpose of manipulating differential equation models becomes possible on a programmable and unified IDEP.100

[0106] Virtual and physical feedback loops Fig. Letter labels "A" through "H" are used to indicate different stages in the product lifecycle. At each stage, IDEP enables a feedback loop in ACP where data originating from physical twins and digital twins is analyzed. This results in a new twin configuration based on design changes. The new twin configuration is stored in the twin configuration set and applied through the application and splice plane, resulting in a modified model file registered on the digital thread.

[0107] The virtual feedback loop 104 begins with a decision 106 to instantiate a new digital twin 122. The DAG of hierarchical tasks 124 enables the automated instantiation of digital twins 12120 within a virtual environment, based on twin configurations applied in process steps 08156 from a set of twin configurations. The digital twin 122 and / or its components are tested in the virtual environment 120, leading to the generation of digital twin performance data 126. Simultaneously, the digital twin 122 and / or its components may be tested and simulated on a model airplane 180 test and simulation performance data was generated using DE software tools 174. The performance data 126 may also be combined with 74 when compared to 152, analyzed in ACP 150, which may lead to the generation and storage of a new twin configuration. Finally, the decision to instantiate the digital twin from the new twin configuration completes the virtual feedback loop 104.

[0108] The physical feedback loop 102 begins with a decision 106 to materialize a new physical twin 132. The physical twin 132 may be instantiated in the physical environment from a model file of a model machine 301 associated with a twin configuration applied from a set of twin configurations 80156. The physical twin 132 and / or its components are tested in the physical environment 132, which leads to the generation of sensory data from physical twin sensors 134 and environmental sensors 36130 located in the physical environment. This sensory data is combined with data from an external database to obtain processed sensory data 144. In one example, temperature measurements from environmental sensors placed in the physical environment are completed, adjusted (e.g., shifted), and / or calibrated using data from an external temperature database.

[0109] Data from physical twin sensors 134 can be directly added to the model file of the model machine 1 by the DE software tool used in the design process of the physical twin 80132. Alternatively, physical twin sensor data 162 can be added to the digital thread associated with the physical twin 132 directly via the application plane 160. Furthermore, processed sensory data 144 can be integrated into IDEP 100 directly via the application plane 160. For example, processed sensory data 144 can be sent to ACP 1 for analysis 50 minutes, which may lead to the generation and storage of a new twin configuration. The final decision to instantiate the physical twin from the new twin configuration completes the physical feedback loop 102.

[0110] At each stage of the product lifecycle, from A to H, the system labels one twin configuration as the current design reference, which we will refer to here as the “authoritative twin” or “authoritative reference.” The authoritative twin represents the design configuration that best responds to the actual conditions (i.e., the root truth). U.S. Provisional Patent Application No. 63 / 470,870 (Case No.: IST-03.001P) provides a more complete description of the authoritative twin and its determination, which is incorporated here in full as a reference.

[0111] By accelerating the feedback loop from sensor data and expert recommendations, the system updates the digital twin122 to reflect the latest design changes. This update process may involve the engineering team analyzing feedback154 and implementing changes through IDEP100, or automated changes being enabled by IDEP100. One of these updates to the digital twin122 is generated by programmed algorithms or AI modules. This iterative update process continues until the digital twin122 is 132 years old, the real twin132 is synchronized, and the product performance has achieved the desired goals. During the IDEP100 period, IDEP100 itself does not designate authoritative references for the digital and physical twins, but provides configurable mechanisms such as policies, algorithms, voting schemas, and statistical support, where agents designate a new digital twin as the authoritative digital twin or designate when the physical twin is the authoritative source of truth.

[0112] If significant design improvements are made, a new physical twin prototype may be built based on the updated digital twin. This new prototype undergoes further testing and validation to ensure that the product's performance and design meet the project's objectives.

[0113] At age 122, where a digital twin once existed, 132 actual twins have been verified and optimized, and the product is ready for production. The digital thread connecting all stages of development is queryable through the splice plane for verification and documentation required to meet verification requirements. The use of model splicing and leveraging the feedback architecture shown in the diagram improves the overall efficiency of product innovation.

[0114] Interconnected DE platforms and product lifecycle As shown in the diagram, 1. The letter labels "A" through "H" indicate the following key steps in the product lifecycle, according to some examples of the present invention: A. Digital models reside within the customer's environment. Products may originally be represented by model files accessible through software tools installed within the customer's environment. Model Airplane 180 includes all model files (e.g., 182) associated with the product. B. Design preparation steps in the digital domain: The splice plane 170 is generated from the DE model file via model splicing (e.g., 172). Model splicing enables the integration and sharing of DE model files within a single platform, as described in detail with reference to Figs 7 to 9. C. To implement linked thread products as needed between model splices, model splices are linked via scripts in the application plane.160. Digital twin 1 can generate product features with Engelob designs from the application plane.1 Operation in a virtual environment is 60120.1. The complete twin configuration of the generated digital twin is stored in the twin configuration set 1 located in the Analysis and Control Plane (ACP).56150.1. Digital twin features or parts 122 can be simulated on the model machine 180, with performance data 174, accessed via the splice plane 170.1 D. Finalizing "As Designed": Performance data from the digital twin (26122) or simulation performance data (74180) achieved in the model aircraft and accessible through model splicing are collected and sent to the ACP (Analysis Program) for analysis (50). Performance data from different versions of the digital twin (122) can be compared in the engine (152) from the design requirements (152). Analyzing these differences may lead to the generation of new twin configurations (156) stored in the twin configuration set (156). Each twin configuration (156) in the twin configuration set is applicable in the application plane (160) and splice plane (108) via process step (101) to instantiate the corresponding digital twin (701). Multiple digital twins can be generated and tested sequentially or simultaneously against the design requirements through the comparison engine (152) and analysis module (154). Verification and validation tools can be run on various digital twin iterations. E. Finalizing the "As-Manufactured": The digital twin 122, which once met the design requirements, is the corresponding physical twin 132 prototype, which can be instantiated from the spliced ​​model file (e.g., 172). Sensor data originating from the physical twin 134, or from within the physical environment 136, can be collected in combination with other external data 142 (e.g., sensor data from other physical environments). The resulting processed sensory data 144 can be sent to the analysis and control plane 150, compared with performance data 150 from the digital twin or simulation 26 (e.g., 174), further linked to the digital twin 122, with 32 iterations filling the set of 1 twin configurations 156. The processed sensory data 144 can also be mapped to digital threads (e.g., 164) and model seams (e.g., 172). The tested physical twin 132 governs the application plane 160. F. Finalizing the "Aspread": Once the manufacturing process for each component is complete, the next step, whether as a digital twin or a physical twin, is the final configuration of the assembly. This involves creating a digital representation of the assembly to meet specified requirements. The digital assembly takes into account the dimensions and tolerances of the parts "at the time of manufacture." To verify the feasibility of the digital assembly, tests are performed using measurement data obtained from the physical assembly and its individual components. The measurement data from the physical components becomes the authoritative standard for the digital assembly, ensuring consistency with the actual configuration. The digital assembly is compared to the actual physical assembly requirements to verify the assembled configuration. Subsequently, the digital assembly tests and configurations become the authoritative reference material for instructions to guide the physical assembly process and ensure accurate reproduction. The components in IDEP 1 above may be used in the assembly process. Its authoritative version, the digital twin 122, ultimately captures the details of the physical assembly accurately, enabling comprehensive analysis and control in subsequent stages of the process. G. Finalizing the "Operating Condition": To evaluate the performance of physical assemblies and their individual components, multiple digital twins can be generated as needed. These digital twins are created based on specific performance metrics and function as virtual replicas of the physical system. The digital twins are continuously updated and improved in real time using operational data (e.g., collected from performance monitoring of physical assemblies and their components). This data includes, but is not limited to, processed sensory data, performance metrics, and other relevant information. By incorporating this real-time operational data, the digital twin is created that synchronizes with the actual system and accurately represents its operational performance. Changes and improvements observed by sensory data during actual assembly operation are reflected in the DE model within the digital twin and recorded in the twin configuration set. This ensures that the digital twin remains up-to-date and consistent with the state of the physical system. H. Predictive Analytics / Future Performance: The design process can be iteratively continued in a virtual environment. There are 22 different configurations when one product is operated, from 120 to a new digital twin. Multiple digital twins are created, allowing for the evaluation of the future performance of physical assemblies and their components based on specific performance metrics. Simulations are performed using various control strategies to evaluate their impact on performance targets and costs. The results of these simulations help determine which control strategies should be implemented (e.g., tail volume coefficient and sideslip angle for aircraft products). Digital twin DE models (e.g., 182) are continuously updated and improved using the latest sensor data, control strategies, and performance metrics to enhance predictive accuracy. This iterative process allows digital twins (e.g., 122, 156) to provide reliable predictions of future performance and support informed decision-making.

[0115] The hardware components that make up IDEP (e.g., servers, computing devices, storage devices, network links) may be centralized or distributed across multiple entities, including one or more DE service providers and DE clients, as will be further discussed in the context of the diagram. 3 and 4. Diagrams illustrate various possible configurations for instanced DE platforms in a customer's physical system and information technology (IT) environment, typically a firewall-protected virtual private cloud (VPC).

[0116] Digital documents through live digital objects The method and system described here leverage all the features of IDMP shown in the diagram to enable the updating and generation of digital documents. 1. In the diagram, 1. The IDMP virtual feedback loop 104 enables the scripting of program code within the digital thread for the creation, storage, and updating of 1 digital twins 122 and twin configuration 156. Similarly, the IDMP virtual feedback loop 104 also enables the scripting of program code within the digital thread 162 for the creation, storage, and updating of digital documents. This makes it possible to create and maintain so-called live digital objects.

[0117] Live digital objects are closer to digital twins than traditional static documents, constructed through digital threads, and continuously updated to reflect the latest changes within a particular twin configuration. In particular, authoritative / trusted live digital objects are configured to reflect the latest authoritative / trusted twin configuration. Specifically, a live digital object is a digital object that (1) contains digital artifacts extracted from a digital model through a model representation (e.g., model splice), and (2) in which modifications to the digital artifacts appear in the live digital object within a predetermined delay. In various forms, updates are virtually real-time or near real-time.

[0118] Live digital objects can use a document interface to generate live digital documents or live documents. Live digital documents can retrieve data from multiple model files. Therefore, preliminary design reviews may take the form of live digital documents.

[0119] Live digital objects can also use dashboard interfaces to create live digital boards, or live boards. In some implementations, a live digital board might display one or more documents and one or more applications on a 2D screen rendered on a multimodal interface modality such as a 2D display, a 2.5D display, or a semi-immersive or fully immersive 3D display. A live digital board can combine multiple documents through VR / AR or conversational interfaces into a 2D or 2.5D board / screen format. For example, a live board might combine multiple model files from CAD software into a collaboration chat room rendered on a 2D display (conventional display), a 2.5D display, or a 3D semi-immersive or fully immersive display. In some implementations, a live board combines multiple view screens.

[0120] Finally, live digital objects can take the form of live digital spaces (or live spaces), 3D virtual environments, or augmented environments. In some implementations, a live digital space displays one or more documents and one or more other applications in a virtual space and renders them through a 3D spatial display. In a live digital space, multiple documents can be combined into a 3D spatial representation through VR / AR or conversational interfaces. For example, in a live space, multiple 3D model files from CAD software might be displayed combined with a collaboration chat room in a 3D semi-immersive or fully immersive spatial display.

[0121] Live digital objects can be stored and accessed through IDMP. Specifically, live digital objects may be used to provide background context for a particular digital thread, and may also be specialized for displaying and organizing artifacts associated with that digital thread, as described in this document.

[0122] Therefore, live digital objects are called magical tools (i.e., live documents are sometimes referred to as "magic documents," live boards as "magic boards," and live spaces as "magic spaces") because changes implemented within the twin configuration (e.g., through modifications to model files) can instantly appear in the relevant data fields of the live digital object. Similarly, authoritative / trustworthy live digital objects are also called authoritative / trustworthy magical objects because they continuously reflect data from the authoritative twin and therefore always represent an authoritative source of truth.

[0123] Given the enormous amount of data and potential modifications that occur during a product's lifecycle, scripts implementing live digital objects can set a predetermined maximum delay between modifications to model files (e.g., correction of digital artifacts) and the execution of corresponding changes within the live digital object. Furthermore, for similar reasons, scripts implementing live digital objects may be limited to specific subsets of model files within the digital twin or system, reflecting only changes to key parameters or settings within the digital twin or system.

[0124] "Printing" live digital documents and boards corresponds to generating frozen (i.e., static) timestamped versions of live digital documents and boards. Therefore, "printing"—for live digital documents and boards—is equivalent to "instantiation" of a digital twin. Similarly, "printing" live digital space is also envisioned, resulting in a frozen 3D representation of a particular system or digital thread.

[0125] In one embodiment of the present invention, an IDMP script (e.g., an IDMP application) can access model data through one or more model splices or digital document templates, create and update live digital objects, and dynamically update live digital objects using a software-defined digital thread on the IDMP platform. In such an implementation, the IDMP script can dynamically receive user interactions. If the user updates data for a model or specific parameter settings (e.g., digital artifacts), the IDMP script may dynamically propagate the user's updates to the live digital object through the corresponding digital thread.

[0126] As another example of the present invention, an IDEP script can instantiate a DE document with specifications sufficient to generate a physical twin. In such an implementation, the IDEP script can receive a digital twin configuration of the physical twin, generate a live digital object associated with that configuration, receive a predetermined timestamp, and generate a printed DE document (i.e., a static, timestamped version of the live digital object with the predetermined timestamp). Such an operation is referred to as "printing the digital twin."

[0127] As another example of the present invention, an IDEP script can instantiate (i.e., "print") a DE document that specifies the updated digital twin when it detects an update. In such an implementation, the IDEP script may detect modifications to the DE model or associated digital threads. Upon detecting a modification, the IDEP script may update the relevant data fields and sections of the live DE document based on the detected modification, and generate a printed DE document containing the updated relevant data fields and sections based on the live DE document, which is always being updated.

[0128] In various implementations, a software-defined digital thread may be associated with an auxiliary magic document (or "magic document") that contains real-time updates of one or more core parameters of the digital thread. In one example, the magic document contains key parameters that describe the implementation of the user's intent. For example, in one example, the magic document for a digital thread might contain key data points or key orchestration script examples that demonstrate the user's intent (e.g., "increase the drone's wingspan by 1%"). In another example, a script-generating ML model takes pseudocode or detailed user instructions derived from the user's intent as input and is trained on the digital threads and documents from previous IDEPs. In addition to generating digital threads containing orchestration scripts and comments, the script-generating ML model is configured to generate a magic document that describes how the generated digital threads correspond to the user's intent.

[0129] In some implementations, user interaction with the DE model, modifications to the DE model, or modifications to associated digital threads may occur through a push configuration. In a push configuration, the model splicer and digital thread scripts send any relevant updates immediately or within a specified maximum delay time. In other implementations, user interaction with the DE model, modifications to the DE model, or modifications to associated digital threads may occur through a pull configuration. The model splicer and digital thread scripts flag recent changes, and the IDEP script queries the relevant DE model (through the model splice) or related digital thread for flagged modifications. In these implementations, the IDEP script can extract modification information from the modified DE model (through the model splice) or modified digital thread and update the live DE document. In yet another implementation, user interaction with the DE model, modifications to the DE model, and modifications to associated digital threads may occur through a pull configuration. In this pull configuration, the IDEP script periodically checks the associated DE model (through model splices) and associated digital threads, and periodically compares the data in the live DE document with the extracted model and digital thread data to check for corrected data fields. These implementations allow the IDEP script to update the live DE document with the corrected data.

[0130] Dynamic document update Some of the examples discussed here relate to document creation, updating, and document management (e.g., review). As mentioned earlier, some implementations of the system allow for dynamic updates of software-defined digital threads and associated documents within the IDEP platform.

[0131] A one-time operation is proposed that uses an ML engine with model data and templates to create and update documents almost instantly. Furthermore, the digital engineering platform dynamically interacts with the user. When the user interacts with the system and updates data for the model or specific parameter settings, these changes may propagate to corresponding digital threads and related documents. The AI ​​architectures involved include large-scale language models (LLMs) that are locally instantiated for data security reasons, as well as non-LLM approaches (e.g., NLP-based) that create, update, or predict documents in the form of sentences, paragraphs, or entire documents. At the same time, attempting to update the entire digital thread with every update would be extremely slow and could pose a security risk to the system. Therefore, it may be more efficient to generate live DE documents that are updated within the maximum latency based on a portion of the system's DE model.

[0132] Interconnected Digital Engineering and Certification Ecosystem Figure 2 shows an exemplary implementation of IDEP as an interconnected digital engineering (DE) and authentication ecosystem and an exemplary digital authentication product based on several examples of the present invention. The interconnected data and authentication ecosystem can be considered a specific instance or implementation of IDEP as shown in Figure 1. IDEP is also called the "DE metaverse".

[0133] The Interconnected Data and Certification Ecosystem 200 is a computer-based system that links models and simulation tools with relevant requirements to fulfill the purposes of verification, validation, and certification. Verification refers to the method of evaluating whether a product, service, or system meets specified requirements and is suitable for its purpose. For example, in the aerospace industry, the verification process includes testing whether aircraft components can withstand the forces and conditions encountered during flight. Verification also includes external checks against customer and stakeholder needs. Validation refers to the method of evaluating whether the overall performance of a product, service, or system is suitable for its intended use, compliance with regulatory requirements, and ability to meet the needs of intended users. Verification also includes internal matching against specifications and regulations. The Interconnected Data and Certification Ecosystem 200, as revealed here, aims to connect and bridge numerous different DE tools and models from multiple engineering disciplines and fields, but from separate organizations that want to share models with each other but do not have other interactions. In various implementations, the system implements a robust, scalable, and efficient DE model collaboration platform, with extensible model splices containing data structures and associated functionalities to support a wide range of distributed DE model types and DE tools. It also features an application layer for linking or connecting DE models via APIs, and a digital thread for connecting, collaborating on, and sharing live engineering model files. Furthermore, it includes digital document management to assist in creating engineering and certification documents suitable for verification and validation (V&V) purposes, and AI assistance for the functionality of the aforementioned system components.

[0134] More specifically, it refers to figs.2 Examples of interconnected DE and certification ecosystems and examples of digitally certified products212A, 212B, and 212C (collectively referred to as digitally certified products)2 For example, in some implementations, a digitally certified product212A may be an unmanned aerial vehicle (UAV) or other aircraft that is a digitally certified product212B may be a drug or other chemical or biological compound that is a digitally certified product212C may be a process such as a manufacturing process. Generally, a digitally certified product212 includes any product, process, or solution that can be developed, tested, or certified (partially or completely) using DE tools (e.g.)202. In some implementations, a digitally certified product212 is not limited to physical products but can also include non-physical products such as methodologies, processes, and software. Physical and physically interacting systems often require multiple DE tools to assess the conformity of common V&V products due to modeling and simulation (M&S) needs, while many complex non-physical systems may also require multiple DE tools for product development, testing, and certification. Given this, other possibilities for digital certification products are also recognized as part of this technology. By including regulatory and certification standards, compliance, calculations, and testing (e.g., for product and solution development, testing, and certification), users can directly integrate relevant regulatory and certification standards, compliance, calculations, and test data into their DE workflows. These regulatory and certification standards, compliance, calculations, and testing are sometimes referred to as “common verification and validation (V&V) products” in this document.

[0135] Digital Authentication Products 2, Figure 12.2 may be designed and / or authenticated using an interconnected data and authentication ecosystem 200. The interconnected data and authentication ecosystem 200 may include user devices 206A, APIs 206B, or other similar human-to-machine or machine-to-machine communication interfaces operated by the user. Users may be humans 2 with varying skill levels 04, or artificial users 200 to APIs 206B, such as algorithms, artificial intelligence, or other software that interfaces with the ecosystem. The ecosystem 200 may further include computing and control systems 208 ("Computing Systems" 2 hereafter 08) which were connected to or included data storage units 218, artificial intelligence (AI) engines 220, and application and service layers 222. In some implementations, the artificial intelligence (AI) engine 220 is a machine learning (ML) engine. The reference to “machine learning engine” 220 or “ML” engine 220 is more generally referred to as “artificial intelligence (AI) engine” 220, and can be extended to “artificial intelligence (AI) engine” 220. For clarity, users selected from a variety of potential human or artificial users will be simply referred to here as “users” 204. In some implementations, the computing system 208 may be a centralized computing system. In some implementations, the computing system 208 may be a distributed computing system. In some cases, users 204 can be considered part of the ecosystem 200, in other implementations, users 204 can be considered separately from the ecosystem 200. The ecosystem 200 may include one or more DE tools 2, for example, data analysis tools 202A, computer-aided design (CAD) and finite element analysis (FEA) tools 202B, simulation tools 202C, drug modeling and simulation (M&S) tools 202D-202E, manufacturing M&S tools 202F-202G, etc.Ecosystem 200 may also include a repository of common V&V products. 2 regulatory standards such as 10210A-210F are medical standards for UAV development and certification. 210G (e.g., CE marking (Europe), FCC Declaration of Conformity (USA), IECEE CB scheme (Europe, North America, parts of Asia, parts of Australia), CDSCO (India), FDA (USA, etc.), medical certification regulations 210H (e.g., ISO 13485, ISO 14971, ISO 9001, ISO 62304, ISO 10993, ISO 15223, ISO 11135, ISO 11137, ISO 11607, IEC 60601, etc.) and manufacturing standards 210I (e.g., ISO 9001, ISO 9013, ISO 10204, EN 1090, ISO (e.g., 14004) and Manufacturing Certification Regulation 210J (e.g., General Conformity Certification (GCC)).

[0136] As shown in the diagram, 2. The computing system 208 is centrally located within the architecture and configured to communicate with user devices (e.g., receiving and sending data) 206A or API 206B are APIs or DE tools associated with artificial users 2APIs or obtained through Software Development Kits (SDKs) 214, and through the repository of common V&V products 2API / SDK interfaces 10216. For example, the computing system 208 can be configured to communicate with user devices 206A and / or API 206B, which are used to send and receive data corresponding to design prototypes, user information (e.g., user authentication), engineering-related inputs and outputs associated with DE tools 202, digitized common V&V products, product design evaluation, user instructions (e.g., search requests, data processing instructions, etc.). The computing system 208 can also be configured to communicate with one or more DE tools 202, which send engineering-related inputs for the execution of analyses, models, simulations, tests, etc., and receive engineering-related outputs associated with the results. The computing system 208 can also be configured to communicate with a repository of common V&V products 210, which is used to retrieve data corresponding to one or more digitized common V&V products 210 or to upload new common V&V products received from users 204 to the common V&V product repository 210. All communications can be securely transmitted and backed up in a manner based on zero-trust security, for example. In some implementations, the ecosystem's computing system may work with regulatory and certification authorities (e.g., through websites operated by the authorities) to retrieve digitized common V&V products published by regulatory authorities that may be relevant to products designed by the user. In some implementations, users may also digitize common V&V products and upload them to the ecosystem.

[0137] The computing and control system 208 can process and store received data, perform analysis and control functions, and in some implementations may also access a machine learning engine 220 and / or the application and service layers 222, to identify useful insights based on the data, as will be further discussed here. The central placement of the computing system 208 within the ecosystem architecture offers many advantages, including reducing the technical complexity associated with the integration of various DE tools, improving the user's product development experience 204; intelligently connecting common V&V products such as standards 210A-210F to DE tools 202 is most useful for meeting the requirements associated with common V&V products; and enabling the monitoring, storage, and analysis of various data flowing between elements of the ecosystem throughout the entire product development process. In some implementations, the data flowing through the computer system, and in some cases stored 208, can also be used for preventing security breaches and implementing data quality control. Similarly, any analysis and control functions performed through the computing system 208 may be tracked for auditability and traceability considerations.

[0138] Referring to the specific example shown in the figure: 2. User 204 may leverage the DE and certification ecosystem to manufacture a digitally certified UAV 212B. For example, user 204 may primarily be involved in certification that the UAV meets the requirements of a specific regulatory standard 210E concerning failure conditions for unmanned aerial vehicles (e.g., "MIL-HDBK 516C 4.1.4 - Failure Conditions"). In this use scenario, user 204 may develop a digital prototype of the unmanned aerial vehicle on user device 206A or use API 206B and send prototype data (e.g., as either a CAD file or an MBSE file) to the computer system 208. Along with the prototype data, user 204 may send additional data via user device 206A, including a display of V&V products commonly used by the user 204 who is interested in product certification (e.g., regulatory standards) 210E), user authentication information 208 to access one or more functions of the computer system, and / or instructions 202 to run one or more digital models, tests, and simulations using part of the DE tools.

[0139] Refer to another example shown in the diagram. 2. User 204 can leverage the DE and certification ecosystem to produce digitally certified pharmaceuticals, chemical compounds, or biologics 212A. For example, user 204 may primarily be involved in the certification of pharmaceuticals, chemical compounds, or biologics 212A, which refers to meeting the requirements of specific medical standards 210G and medical certification regulations 210. In this usage scenario, the user 204 can develop digital prototypes of drugs, chemical compounds, or biological products on the user device 206A or use the API 206B, and can send prototype data (e.g., as molecular modeling files) to the computer system 208. Along with the prototype data, the user 204 can send through the user device 206A, additional data 204 including a display of common V&V products used by the user that the user is interested in product certification (e.g., medical standards 210G and medical certification regulations 210H), user authentication information 208 to access one or more functions of the computer system, and / or instructions 202 to run one or more digital models, tests, and simulations using parts of the DE tools (e.g., pharmaceutical M&S tools) 202D-202E).

[0140] Referencing another example shown in the figure, 2, User 204 can leverage the digital engineering and certification ecosystem to produce digitally certified manufacturing processes 212C. For example, User 204 may primarily be involved in the certification of a manufacturing process 212C as meeting the requirements of specific manufacturing standards 210I and Manufacturing Certification Regulations 210J. In this use scenario, User 204 can develop a digital prototype of the manufacturing process on a user device 206A or use an API 206B and send the prototype data to a computer system 208. Along with the prototype data, User 204 can send through the user device 206A, additional data including a display of common V&V products used by the user 204 who is interested in the certification of (e.g., manufacturing standards 210I and Manufacturing Certification Regulations 210J), user authentication information 208 to access one or more functions of the computer system, and / or instructions 202 to perform one or more digital models, tests, and simulations using parts of DE tools (e.g., manufacturing of M&S tools) 202F-202G).

[0141] In any of the examples described above, the computing system 208 can receive data transmitted from the user device 206A and / or use API 206B to process the data in order to evaluate common V&V products (e.g., regulatory standards 210E, medical standards 210G, medical certification rules 210H, manufacturing standards 210I, manufacturing certification rules 210J, etc.) that are met by the user's digital prototype in the context of analysis and control as shown in Figure 1, 501. This may include, for example, communication with a repository of common V&V products via API / SDK 2, 10216: retrieving common V&V products of interest and processing regulatory and / or certification data related to the common V&V products to identify requirements for the UAV prototype; pharmaceuticals, chemical compounds, or biological prototypes; prototypes of manufacturing processes, etc. In some implementations, a repository of common V&V products is provided,210 hosted by regulatory authorities and / or certification bodies (or other third parties), and the use of an API / SDK is required to retrieve regulatory and / or certification data,216 interacting with one or more data resources managed by regulatory authorities and / or certification bodies (or other third parties). In some implementations, regulatory and certification data may also be provided directly by the user,2 via user devices,04206A and / or API206B (e.g., along with prototype data).

[0142] To evaluate whether a user's digital prototype meets common V&V products, processing of prototype data received from the user device is also included. API 206A or API 206B is used to determine whether one or more requirements are actually met. In some implementations, the computing system 208 can process the prototype data directly on the computing system, including one or more plug-ins or local applications. For example, applications of model splicing and digital threading will be discussed in detail with reference to the diagram below. In some implementations, the computing system may simply preprocess the received prototype data (e.g., to derive inputs for a DE tool) using API 202, and instructions and input data can be sent to a part of the DE tool 202 via API / SDK2 for further processing.

[0143] Not all DE tools 202 are necessarily required to meet certain regulatory and / or certification standards. For example, in the UAV example shown in the figure, the calculation system 208 may only determine that data analysis tools 202A and finite element analysis tools 202B are required to meet regulatory standards and 10E to indicate failure conditions. In the drug, chemical compound, or biological example shown in the figure, the calculation system 208 may only determine that drug M&S tools 202D-202E must meet medical standards 210G and medical certification regulations 210J. In the example of a manufacturing process shown in the figure, the calculation system 208 may only determine that M&S ​​tools 202F-202G are required to meet manufacturing standards 210I and manufacturing certification regulations 210J. In other implementations, the user 204 themselves can identify a subset of specific DE tools that should be used to meet common V&V products, for example, if 204 is a qualified professional (SME). In other implementations, user 204 may input 208 into the calculation system to satisfy the common V&V product of the recommended DE tool 2, and the calculation system 208 can be recommended to the user 204 for final approval by the user of the modified subset 2 of the micro-equation tool 2, if that user is a qualified SME. Having been identified through a part of the DE tool 202, the calculation system 208 can send instructions and input data to the identified subset of the DE tool 202 used to run one or more models, tests, and simulations. The results of these models, tests, and simulations (or "engineering-related data output" or "digital artifacts") can be sent to and received from the computer system 208. In further implementations, user 204 can input the following required DE tools: 02F210I to satisfy the common V&V product, and the computing system 208 can determine that another differential equation, such as 102G, is also required to satisfy the common V&V product. The computing system can then send instructions and input data to both DE tools (e.g., 202F and 202F). The outputs of these DE tools can be sent and received on the computer system 208. In some cases, the input data submitted to one of the DE tools (e.g., 202G) can be derived (e.g., by the computing system 208) from the output of another differential equation tool (e.g., 202F).

[0144] After receiving engineering-related data output and digital artifacts from the DE tool 202, the computing system 208 then processes the received engineering-related data output and evaluates whether the requirements identified in common V&V products (e.g., regulatory standards 210E, medical standards 2110G, medical certification rules 210H, manufacturing standards 210I, manufacturing certification rules 210J, etc.) are satisfied. For example, applications and services 222 can provide instructions regarding verification and the orchestration of verification activities. In some implementations, the computing system 208 generates a report summarizing the evaluation results and can send that report to the device 206A or API 206B (for user review) 204. If all requirements are met, the prototype is certified and becomes a digitally certified product 212 (e.g., digitally certified pharmaceuticals, chemical compounds, or biologics) 212A; digitally certified UAVs 212B; digitally certified manufacturing processes 212C, etc.). However, if some regulatory requirements are not met, additional action may be required on the user's side.204 was done to certify product prototypes. Some implementations may include recommendations for these additional steps in reports sent to users (e.g., one or more design changes, one or more suggested replacements of components in existing design solutions, one or more suggested adjustments to inputs for models, tests, and simulations).202 If the requirements for common V&V products are partially met or exceed the collective capabilities of distributed engineering tools, the computing system208 may provide users with reports recommending partial certification, compliance, or performance of common V&V products (e.g., digital certification of subsystems or subprocesses of prototypes).204 The process for generating recommendations to users204 is described in more detail below.

[0145] For the review of the report, the user can make local design changes to the digital prototype or send one or more instructions to the computer system via the user device 08206A or API206B. These instructions include, for example, instructions to the computer system 208. To re-evaluate the updated prototype design, one or more different DE tools are used 202 for the evaluation process, or to correct the input to the DE tool 202. The computing system 208 receives the user instructions and performs one or more additional data operations according to these instructions, and the user 204 receives an updated report 204. Through this iterative process, the user 204 can leverage an interconnected digital engineering and certification ecosystem to design prototypes (e.g., UAV prototypes, pharmaceutical prototypes, manufacturing process prototypes, etc.) for common V&V products and ultimately obtain certification (e.g., providing certification compliance information). Importantly, because all these steps are performed in the digital world (e.g., digital prototypes, digital models / tests / simulations, digital certifications, etc.), significant time, cost, and material savings are possible compared to processes involving the physical prototyping, evaluation, and certification of similar UAVs, pharmaceuticals, and manufacturing processes. If the requirements related to the common V&V product are partially met or exceed the overall capabilities of the DE tool,202 the computing system 208 may provide the user with a report recommending partial certification, compliance, or implementation of some of the common V&V products (e.g., digital certification of subsystems or subprocesses of prototypes).

[0146] The above example focuses on a single user's use of an interconnected digital engineering and certification ecosystem, but additional benefits of the ecosystem can be gained through repeated use by multiple users. As mentioned earlier, the central location of the computing system 208 enables the computing system within the ecosystem architecture 208 to monitor and store various data flows within the ecosystem. Therefore, as the number of users utilizing the ecosystem for digital product development increases, data associated with each use of the ecosystem can be stored (e.g., in storage) 218), traced (e.g., with metadata), and analyzed to gain various insights, further automating the digital product development process and making it easier for non-experts to understand the digital product development process.

[0147] In fact, some implementations allow for user authentication information to be used on a per-user basis, which can indicate the user's skill level and control the amount of automated assistance provided. For example, non-experts may be allowed to use the ecosystem to browse off-the-shelf designs and solutions and use DE tools, following predetermined workflows guided by specific default parameters or automated assistants throughout the product development process. On the other hand, more skilled users may receive automated assistance, but they may also have more opportunities to override default or recommended workflows and settings.

[0148] In some implementations, the computing system 208 can host applications and services 222 that automate or partially automate components of common V&V products, including expected or common data transmission from users, data transmission components 204; data exchange including expected or common interfaces and / or interface components between various DE tools 202; if machine learning (ML) models are implemented on the computing system, this also includes expected or common interfaces and data exchange, interface components 208 (e.g., models trained or implemented by the ML engine) 220; and expected or common interfaces and / or data exchange between applications and services (e.g., within the application and service layers) 222). In some implementations, it is possible to aggregate multiple usage data (or parts thereof) from the ecosystem to create training datasets. For example, usage records 217 collected through a computer system 208 may be deidentified or anonymized before being added to the training set. These usage records may include model parameters and metadata, tool settings, general V&V product matching for specific models and tools, interactions with user systems including inputs and behaviors, and other user-defined or system-defined settings and decisions. For example, an exemplary deidentified usage record may consist of a combination of a specific DE tool, a specific target metric, a deviation of a specific quantity, and a corresponding specific user update to the DE model under this configuration. Another exemplary deidentified usage record may consist of a subset of user-identified DE tools 2 that should be used to satisfy a common V&V product 02.

[0149] This training dataset can be used to train ML models (e.g., using an ML engine)220) to learn the steps and actions of the certification process and perform a variety of tasks such as selecting which DE tools to use to satisfy specific common V&V products; identifying specific models, tests, and simulations (including inputs) to be performed using DE tools202; identifying common V&V products to consider for specific types of products; identifying one or more actions to be recommended for users2 to take in response to failures of regulatory requirements04; estimating the sensitivity of models / tests / simulations to specific inputs; etc. The output of the trained ML model can be used to implement various functions of the interconnected digital engineering and certification ecosystem, including automatically suggesting inputs (e.g., inputs to DE tools)202) previously entered inputs, predicting the time and cost required for product development, predicting the results of sensitivity analysis, and even suggesting design changes, original designs, or alternatives (e.g., by auxiliary or generative AI) to overcome one or more requirements (e.g., regulatory or certification requirements) related to common V&V products. In some implementations, if sufficient training data is available, the ML engine can independently generate new designs, models, simulations, tests, common V&V products, or digital threads based on data collected from multiple uses of the ecosystem. Furthermore, new designs, models, simulations, tests, common V&V products, and digital threads generated by the ML engine are also included. After approval and adjustment by 2 users, these are added to the training set for further fine-tuning of the machine learning algorithm in a reinforcement learning setup.

[0150] This will be discussed in the context of Figs.7 Destination9, the aforementioned collection of training datasets, and the training of ML and AI modules, including the ML engine220, can be made possible by model splicing techniques. The model splicing described here enables the scripting of DE model operations encompassing different DE tools into a corpus of prescriptive program code, enabling code-defined digital threading of a wide range of DE activities, including DE models across different domains. ML and AI techniques are used to create scripts that perform virtually any DE task and execute any digital thread, enabling programmable, machine learning-enabled, and dynamic changes to DE model files, digital threads, and ultimately digital or physical twins throughout the entire product lifecycle. For example, in the embodiment shown in Figs2, the ML engine220 can manage or orchestrate interactions between spliced ​​DE models, DE tools, and common V&V products (e.g., DE requirements) based on digital thread options that respond to user intent and input. Sample DE tasks that the ML engine can perform include, but are not limited to, (1) aligning models and analyses to authentication lifecycle requirement steps, (2) determining appropriate fidelity for each model to optimize computations, (3) optimizing computational resources for specific tools and models, and (4) optimizing computational resources across multiple models. Performing ML-enabled DE tasks is not limited to authentication and resource optimization, but encompasses the entire DE operating space. Rather, the ML engine 220 can function as an AI multiplexer for the DE platform.

[0151] In addition to saving usage data that enables the development of ML models, past prototype designs and solutions (e.g., previously designed components, systems, models, simulations, and other engineering representations) can also be stored within the ecosystem (e.g., 218 in storage). This allows users to search for others' work and build upon it. For example, users can search for previously designed components, systems, models, simulations, and other engineering representations 204 and / or it is recommended to users 204 to meet one or more requirements related to the common V&V product per computer system 2. Previously designed components, systems, models, simulations, and other engineering representations are available to users 204 and can be used as is or as a starting point for additional modifications. This store, or repository, can be monetized by containing previously designed components, systems, models, simulations, and other engineering representations (whether ultimately certified or not) and creating a marketplace for digital products. This saves time in the digital product development process, inspires users with ideas for alternative designs, and avoids redundant work. And more. In some implementations, data corresponding to previous designs and solutions may only be stored if the user who developed the design or solution chooses to share the data. In some implementations, repositories of past designs and solutions can be containerized for private use within a single company, team, organization, or technology field (e.g., to avoid the unnecessary leakage of sensitive information). In some implementations, user authentication information is associated with the user and can be verified by the computing system to determine whether the designs and solutions stored in the repository are accessible to the user. In some implementations, the use of previously designed components, systems, models, simulations, and other engineering representations may be available only to other users who have paid a usage fee.

[0152] Exemplary IDEP implementation architecture with services and features Figure 3 shows another exemplary implementation illustrating the services and features offered by IDEP in accordance with several implementations of the present invention. Specifically, the exemplary implementation architecture Figure 300 is shown in the figure. To include multiple illustrated components: IDEP enclave 302, cloud services 304, and customer environment 310, optionally including IDEP enclave 316. This exemplary architecture 3 IDEP is designed in accordance with zero-trust security principles and is further designed to support scalability and robust and resilient operation. IDEP enclave 302 and IDEP enclave 316 instantiate IDEP together as shown in Figure 1. Model splicing and splice line implementation 1 In some examples of the present invention 70. An enclave is a collection of independent cloud resources partitioned to be accessible to a single customer (i.e., single-tenant) or market (i.e., multi-tenant), and does not depend on the resources of other enclaves. An exclave is a collection of cloud resources outside of an enclave managed by IDEP, used to perform work for individual customers. Examples of exclaves include virtual machines (VMs) and servers maintained by IDEP, where customers may require DE tools.

[0153] In particular, the IDEP Enclave or DE Platform Enclave 302 serves as the starting point for services offered by IDEP and can be envisioned as a central command and control hub responsible for managing and orchestrating all platform operations. For example, Enclave 302 can be implemented using a computer system as shown in Figure 2 of the interconnected DE and authentication ecosystem. The DE Platform Enclave 302 is designed to integrate both a zero-trust security model and hyperscale capabilities, providing a secure and scalable processing environment tailored to individual customer needs. Zero-trust security features include, but are not limited to, strict access control, algorithmic fairness, and data isolation. Enclave 302 also supports the following ML engines: 20 points for real-time analytics, 20 points for auto-scaling for workload adaptability, and 20 points for interoperability with API-based third-party services. Security and resource optimization are enhanced through multi-tenancy support, role-based access control, and data encryption (quiescein and in transit). The DE Platform Enclave 302 may include one or more of the features described below.

[0154] First, IDEP Enclave 302 is designed in accordance with zero-trust security principles. Specifically, the DE platform's Enclave 302 uses zero-trust principles to ensure that no implicit trust is assumed between any elements, such as the digital model, platform agents, individual users (e.g., users) or actions within the system.204) In other words, no agent can be trusted in principle, and the system can always authenticate or authorize a particular job. This model is further reinforced by a strict access control mechanism, restricting even the administrator team (e.g., a team of individuals associated with the platform provider) to predetermined and limited access to the Enclave resources. To further enhance this robust security posture, data encryption is applied both at rest and in transit, effectively mitigating the risk of unauthorized access and data breaches.

[0155] IDEP Enclave 302 can also be designed to maintain isolation and independence. A key aspect of enclave architecture is the emphasis on fairness and isolation. DE Enclave 302 prohibits cryptographic dependencies from external enclaves and enforces a strong isolation policy. The enclave design also allows for single-tenant and multi-tenant configurations, further enhancing the isolation of data and processes between customers (e.g., users) (306, 204). In addition, DE Enclave 302 is designed with an isolated set of resources, minimizing interdependencies and promoting system efficiency and autonomy.

[0156] The IDEP Enclave 302 can be designed with an even greater emphasis on scalability and adaptability to meet diverse operational requirements. For example, the Enclave 302 combines hyperscale-like characteristics with zero-trust principles to enable scalable growth and effectively handle high-performance workloads.

[0157] IDEP Enclave 302 is designed with a strong emphasis on workflow adaptability, accommodating diverse customer workflows and DE models through rigorous access control mechanisms. This configurability allows for the modularization of various functions, from data ingestion to algorithm execution, while maintaining a zero-trust security posture. The adaptability of Platform 300 makes it highly versatile for a wide range of applications, ensuring consistent performance and robust security.

[0158] The IDEP Enclave 302 can also be designed to enable analytics for robust platform operations. At the heart of the Enclave's operational efficiency is a machine learning engine (e.g., a machine learning engine)220) enabling real-time analytics. This improves decision-making and operational efficiency across the entire platform300. An auto-scaling mechanism is also incorporated, enabling dynamic resource allocation in response to workload demands, further enhancing the platform's responsiveness and efficiency.

[0159] As an exemplary example shown in the figure, 3. IDEP Enclave 302 includes several components, which are described in more detail below.

[0160] The "Monitoring Service Cell" may provide "Monitoring Services" and "Telemetry Services." A cell may refer to a collection of microservices running within a Kubernetes pod, for example. These components focus on maintaining, tracking, and analyzing platform performance, including advanced machine learning capabilities for real-time analysis, to ensure good service delivery. The "Search Service Cell" provides a "Search Service" to help efficiently retrieve information from the DE platform, enhancing overall functionality. The "Log Service Cell" and "Control Plane Service Cell" provide "Log Services," "File Services," and "Job Services," recording and managing operational events and information flows within the platform, playing a crucial role in platform functionality. The "Static Assets Service Cell" provides "Static Services" and may house the user interface, SDK, command line interface (CLI), and platform documentation. The "API Gateway Service Cell" provides "API Gateway Services" and may provide the DE platform API (e.g., API). Platform services also include intermediaries for requests between client applications (e.g., DE tools) 202, repositories of common V&V products 210, etc. In some implementations, the API gateway service cell may receive and respond to requests from agents, such as enclaves of the DE platform, to provide splicing functionality for model splicing 16.

[0161] As shown in the diagram, 3. The DE platform architecture 300 may also include cloud services 3 that provide services that do not interact with customer data but provide services that allow for software modification for the orchestration of DE platform operations 04. In the example implementation, multiple cloud resources provide support and foundational services to the platform. For example, in the embodiment of the DE platform shown in 3 Figure 00, 3. Cloud services 304 include a "Customer Identification and Access Management (IAM) service" that ensures secure and managed access to the platform 300. Cloud services 304 also include a "test service" that tests tools for verifying platform operations. Cloud services 304 may also include an "orchestration service" that controls and manages the lifecycle of containers on the platform 300. Cloud services 304 may also include "artifact services" and "version control and build services" which can be used to manage artifacts generated during the product development process while maintaining the evolution of projects, code, and instances within the system.

[0162] As shown in the diagram, the DE platform architecture 300 may also include a customer environment 310 with the “authoritative source of truth”, customer tools 314, and an optional DE platform enclave 316. The customer environment 310 is where customer data is stored and processed by the DE platform 300 in a zero-trust manner. As previously mentioned, the DE platform enclave 302 focuses on both zero-trust principles and hyperscale-like characteristics to provide a robust and scalable environment for secure handling of critical workloads tailored to the customer’s unique needs. In some cases, there may be a DE platform enclave 316 located within the customer environment 3 responsible for DE tasks and operations, including model splicing and digital threading, to assist the customer 103.

[0163] When a customer is present (e.g., a user) (204), the DE platform is intended to perform DE tasks (e.g., IDEP 100), and typical operations include secure data ingestion and controlled data retrieval. Derived data generated through DE operations, such as updated digital model files or modifications to digital model parameters, may be stored only within the customer environment (310), and the DE platform (300) may provide tools for accessing metadata of the derived data. Here, metadata refers to data that can be viewed without opening the original data and may include version information, timestamps, and access control properties. An example is secure data ingestion (310) that uses zero-trust principles to ensure that customer data is securely uploaded to the customer environment, connected through a pre-validated secure tunnel such as a Secure Socket Layer (SSL) tunnel. This enables direct and secure file transfers to designated cloud storage within the customer environment (e.g., a Simple Storage Service (S3) bucket).310 Examples may also include controlled data retrieval, which controls data access using temporary and pre-authenticated URLs generated with a secure token-based mechanism, minimizing the risk of unauthorized transactions. Examples include immutable derived data, where transformed data generated by operations such as data extraction is securely stored within the customer environment while adhering to zero-trust security protocols.10 Examples may also include tokenization utilities, which deploy a specialized DE platform tool called a "tokenizer" in the customer environment for the secure management of derived metadata in accordance with zero-trust guidelines.310

[0164] The customer environment 310 can interact with other elements of the secure DE platform 300 and has multiple functions that handle data storage and secure integration with the platform 300. For example, one element of the customer environment 310 is the “Authoritative Source of Truth” 312, which is the primary repository of customer data and ensures the integrity and accuracy of the data. Within this is what is called the “Customer Bucket,” where data is securely stored with strict access control and is limited to authorized users and processes through pre-authenticated URL links. This configuration ensures uncompromising data security within the customer environment 310 and provides seamless integration with other elements of the DE platform 300.

[0165] The customer environment 310 may also include additional software tools such as customer tools 3, which are available depending on specific customer requirements 14. For example, the "DE Tool Host" component may handle DE applications necessary for handling customer data. It may also include a command-line interface for DE tools (DET CLI), enabling user-friendly command-line operation of DE tools (e.g., DE Tool) 102). The "DE Platform Agent" ensures smooth communication and management between customer environments 310 and elements of the DE Platform 300. In addition, there are optional DE toolsets to support customer-specific DE workflows. Native DE tools are typically restricted in access by their own licenses or end-user license agreements paid by the customer. The functionality of the IDEP platform invokes native DE tools running within the customer environment 3, thus strictly adhering to the zero-trust principle of system design. Exemplary DE tools include, but are not limited to, proprietary and open-source versions of model-based systems engineering (MBSE) tools, augmented reality (AR) tools, computer-aided design (CAD) tools, data analysis tools, modeling and simulation (M&S) tools, product lifecycle management (PLM) tools, multi-at-trouble trade space tools, simulation engines, requirements modeling tools, electronics modeling tools, test plan modeling tools, cost modeling tools, scheduling tools, supply chain modeling tools, manufacturing modeling tools, cybersecurity modeling tools, or mission effectiveness modeling tools.

[0166] In some cases, we also offer the optional "IDEP Exclave." 316 individuals may be employed within the customer environment. 310 are responsible for supporting the customer's DE tasks and operations, overseeing data processing, and providing hyperscale-like platform performance while strictly adhering to zero-trust principles. IDEP Exclave 316 is maintained by IDEP and runs the customer's DE tools. IDEP Exclave 316 may include a "DE Tool Host" that runs the DE tools and a "DE Platform Agent" required for operation. Here again, native DE tools are typically restricted in access by the customer's own licenses or end-user license agreements. 316 utilities are used to manage proprietary DE tools hosted in the customer environment, for example, to implement model splicing or digital threading capabilities.

[0167] In some implementations, the machine learning (ML) models and artificial intelligence (AI)-assisted approaches described here are adapted to different customer instances of IDEP (see figure). 4) And the availability of training data. For example, pre-trained ML or AI models (e.g., within an IDEP enclave)302) are deployed when there are restrictions on sharing customer data. Another example is when an AI model is deployed federally adjacent to DE agents or DE tools within the customer environment (e.g., within an IDEP enclave316). Yet another example is when an AI model deployed within the customer environment is trained on the other side of a firewall. Yet another example is when a customer may allow the sharing of a subset of metadata from a training database located within an IDEP enclave.

[0168] IDEP Deployment Scenario Figure 4 illustrates several examples of the present invention, showing possible scenarios for instantiating IDEP in a customer's physical system and IT environment. In particular, Figure 4 illustrates various possible configurations for instantiating IDEP ("DE Platform") related to a customer's IT environment and physical system. The IT environment may reside on a firewall-protected virtual private cloud (VPC). The physical system may refer to the physical twin discussed with reference to the figure. 1. In some examples, IDEP 402 may be instantiated as shown in Figure 3. For example, IDEP 402 may also be instantiated on the cloud and may in some cases operate in a Software-as-a-Service (SaaS) configuration. The platform instances in these implementations include software and algorithms, which can be described as follows: 1. External Platform Instance 410: This option presents IDEP as a separate platform instance. This platform interacts with the physical system through the customer's virtual environment or a customer virtual private cloud ("Customer VPC") connected to the physical system. 2. External Platform Instance 420 with Internal Agent: IDEP is instantiated as a separate platform and connected to an internal agent ("DE agent") fully instantiated within the customer's VPC. For example, IDEP might be instantiated as enclave 302, and the DE agent might be instantiated as exclave 316 within the customer's VPC linked to the physical system. 3. External Platform Instances, Internal Agents, and Edge Computing 430 In this scenario, we view the IDEP as a separate instance, connected to an internal DE agent fully instantiated within the Customer VPC, and further linked to edge instances on the physical system ("DE Edge Instances"). The DE agent is nested within the customer environment and has a small edge computing instance connected to the physical system. 4. Edge Instance Connection 440: This option indicates a DE platform directly linked to a DE edge instance on a physical system. The DE platform and physical system are depicted separately, with the edge computing instance connected in the middle, illustrating the data flow. 5. Direct API Connection 450: This deployment scenario shows the DE platform connecting directly to the physical system via API calls. In this depiction, arrows extend directly from the platform sphere to the physical system sphere, illustrating direct interaction through the API. 6. Air-Gapped Platform Instance 460 This scenario demonstrates that IDEP is instantiated as a DE agent on a fully air-gapped or isolated physical system. This platform operates independently of any network or internet connectivity, providing an additional layer of security by eliminating external access points and potential threats. Interaction with the platform in this context takes place directly on the physical system, and data exchange outside the physical system is controlled according to strict security protocols, maintaining an air-gapped environment.

[0169] In these deployment scenarios, IDEP plays a crucial role in bridging the gap between the digital twin established through IDEP and its physical counterpart. Regardless of how IDEP is instantiated, it interacts with physical systems directly or through the customer's virtual environment. The use of edge computing instances in some scenarios demonstrates the trade-off between the need for localized data processing and more precise insights in real-time analytics and digital-physical system management. Furthermore, the platform's ability to connect directly to physical systems via API calls highlights the importance of interoperability to facilitate efficient data exchange between the digital and physical worlds. In all cases, the DE platform operates with robust security measures.

[0170] In some real-world examples, an IDEP deployment to the same physical system may consist of a combination of the deployment scenarios described above. For example, for the same customer, some physical systems might have direct API connectivity to the DE platform (Scenario 5), while others have edge instance connectivity (Scenario 4).

[0171] Multimodal user interface Fig. 590 demonstrates the use of a multimodal user interface, an interconnected DE platform that handles various input and output modalities such as virtual reality (VR), mixed reality (MR), auditory, text, and code. These interfaces are designed to manage the complexity of data streams and decision-making processes, providing decision support such as optional visualization, impact prediction, and specific decision calls. Specifically, the data streams 502 and 504 are processed in the Analysis and Control Plane (ACP) 1 Fig. 50.1. The user interface can receive data streams from physical and virtual feedback loops 102 and 104, and also includes feedback from external experts 114, analysis module 154, and twin configuration set 1ACP 150.

[0172] The multimodal interface shown in Figure 5 is configured to perform all DE tasks and actions described in the context of Figure 5. It accommodates both humans and bots / algorithms, and is achieved by handling the frequency and complexity of data streams, the timescale of decision-making, and the impact of latency. For human decision-makers, the user interface may need to manage inputs and outputs, while in algorithmic decision-making, the user interface may need to present the reasoning and decision analysis to the human user. Examples of human interfaces include dashboard-style interfaces 594, workflow-based interfaces 596, conversational interfaces 598, spatial computer interfaces 592, and code interfaces 599.

[0173] The dashboard-style interface 594 provides a customizable overview of data visualizations, performance metrics, and system health indicators. It enables monitoring of relevant information, section-by-section review of documents, and decision-making based on dynamic data updates and external feedback. Such interfaces are accessible through web browsers and standalone applications on various devices.

[0174] The workflow-based interface 596 guides the decision-making process, presenting relevant data, options, and contextual information at each stage. It integrates external feedback and is designed as a progressive web or mobile application. In the context of alternative tool selection, the workflow-based interface 596 may offer individual tool options at each stage, or combine tool selections across multiple stages to improve accuracy and efficiency in the overall workflow.

[0175] The conversational interface 598 is based on converting various input formats, such as text, prompts, voice, and audiovisual, into input text, and integrating the resulting input text into the DE platform workflow. Output from the DE platform may undergo the reverse process. This enables interoperability with the DE platform, particularly the manipulation of model splices. In the broader context of voice and video input, conversational interfaces may include data sonification, which refers to using voice to represent data, information, and events, and using auditory cues and patterns to convey important information to users, operators, and reviewers. Sonified alerts (e.g., alerts sent audibly through a speaker) are particularly useful when you want to process information quickly without visually focusing on a screen. For example, sonified alerts can be used to notify security analysts of potential threats or breaches.

[0176] According to the latest advance technologies, a "conversational interface" or "conversational user interface" refers to a human-computer interaction model in which users can interact with digital systems through natural language via text or voice. These interfaces leverage advanced natural language processing (NLP), machine learning, and artificial intelligence technologies to mimic human conversation, understand user input, and respond. Conversational interfaces take various forms, such as chatbots, voice assistants, and messaging platforms, enabling users to communicate with systems in everyday language rather than traditional graphical user interface elements. The goal of these interfaces is to leverage the familiar paradigm of conversation to provide a more intuitive, accessible, and personalized user experience, allowing users to perform tasks, obtain information, and operate devices through natural conversation without requiring expertise in complex commands or navigation structures.

[0177] Fig. 5 also shows examples of using the spatial computing interface. 592 and code interface 5 in managing digital twins and physical twins 99. The spatial computing interface enables a more immersive and intuitive user experience and real-time synchronization between digital and physical twins. The code interface allows bots and digital engineers to interact with the DE platform through scripts and code. It also allows for the collection of user preferences, task history, and tool usage patterns, which can be used for the purpose of selecting alternative tools.

[0178] A "spatial interface" or "spatial user interface" refers to a paradigm of user interaction that utilizes three-dimensional space and spatial relationships to present and manipulate digital information. This approach goes beyond traditional 2D graphical user interfaces, incorporating depth, volume, and spatial position to create a more intuitive and immersive user experience. Spatial interfaces leverage technologies such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) to overlay digital content onto the physical world or create entirely virtual environments. These interfaces allow users to interact with digital objects and information as if they were physical entities in space, using natural gestures, body movements, voice and gaze direction, and spatial awareness to navigate, manipulate, and organize content in a way that more closely resembles real-world interactions.

[0179] In the context of multimodal interfaces, "2.5D" (often referred to as 2.5-dimensional) refers to a visual representation that lies between traditional 2D interfaces and full 3D interfaces. It typically involves creating a pseudo-3D effect by adding depth and perspective to 2D elements without rendering a full 3D environment. The 2.5D approach is usually designed to create the illusion of depth and three-dimensionality on flat, two-dimensional displays such as computer monitors, smartphone screens, and tablets, but it can also be used in 3D environments (e.g., overlaying a 2D screen onto a 3D environment). This technique uses techniques such as layering, parallax scrolling, and isometric projection to create the illusion of depth and volume while maintaining the simplicity and familiarity of a 2D interface.

[0180] Digital threads and autonomous data links As mentioned earlier, "digital threads" connect two or more digital engineering (DE) models to achieve traceability throughout the entire system engineering lifecycle, and aim to facilitate collaboration and sharing among individuals performing DE tasks. In digital threads, appropriate outputs from the previous digital model are provided as inputs to the next digital model, enabling the flow of information and processes. In other words, digital threads can be seen as a communication framework and data-driven architecture that connects elements that were previously siloed. This enables the flow of information and actions between digital models.

[0181] Figure 6 illustrates the architecture and inherent complexities of digital threads, following the example shown here. In particular, Figure 6 is a schematic comparing exemplary digital threads of varying complexity in manipulating and connecting DE models, according to several implementations of the present invention. In its most basic sense, a digital thread can "thread" DE models into a simple daisy-chain architecture, where any change in an upstream DE model affects all DE models downstream of the modified DE model. For example, any modification of parameters or processes in differential equation model B results in a change in differential equation model C, which in turn causes a change in differential equation model D. The resulting changes in relations spread downstream in a chain reaction. As another example, Figure 604 illustrates a more complex digital thread where a change in one DE model can affect multiple downstream models. In both Figures 602 and 604, digital threads are represented by directed acyclic graphs (DAGs).

[0182] DAGs are frequently used in various data processing and structuring tasks, such as scheduling tasks and data compression algorithms. In the context of service platform and network complexity, DAGs are sometimes used to represent relationships between different components and services within a platform. In digital threads, each model can depend on the others in different ways. Model A influences models B, C, and D; models B and C influence model E; and models D and E influence model G. Such dependencies are called DAGs, where each node is associated with a component (e.g., a model) and each directed edge represents a dependency.

[0183] A major problem when dealing with interdependent differential equation models is that graph consistency can have polynomial and exponential complexity. Therefore, if a node fails (e.g., if a model becomes unreliable), it can have a cascading effect across the entire digital thread, disrupting the entire design. Furthermore, adding nodes and dependencies to the graph does not increase complexity linearly due to interdependencies between models. If a new model influences or depends on multiple existing models, the increase in graph complexity is multiplicative and can grow exponentially. The multiplicative nature of digital thread consistency is further complicated by the number of interconnected models, which can range from hundreds to thousands. Figure 606 is a partial representation of a real-world digital thread, illustrating the complexity of a digital thread and its multiplicative growth.

[0184] Figures 603, 605, 607, 608 show 6 more specific cases, and 9 exemplary simple digital threads. Figure 607 represents a degenerate digital thread sharing data from a single DE model. Figure 608 represents a model-to-document digital thread that can generate or update a text-based document (e.g., Capability Development Document (CDD)) using data extracted from a single DE model (e.g., system attributes, performance attributes). Figures 603 and 605 are generalized from Figure 608, which represents updating multiple models using data extracted from a single model, or vice versa. Specifically, Figure 605 could represent the dynamic updating of live documents or magic documents discussed in the context of the figure. 1. The logic connecting the DE models shown here is clear: extract data from multiple DE models A, B, and C to update document model D. There is no interaction between the extracted data. Furthermore, Figure 609 shows a special case of a digital thread that loads and extracts data from only a single model A. For example, as discussed in the context of the diagram. Next, we show the input splice function of Model A. You can run 609 to update the model and output the splice function of Model A. As shown below, 609 can create and share digital artifacts. For these special simple threads, IDEP may provide the user with a GUI-based interface for connecting models and executing digital threads. For complex threads, 606 a code-based interface may be required.

[0185] Model splicing for digital threading and digital twin generation As disclosed herein, model splicing encapsulates and compartmentalizes digital engineering (DE) model data and the manipulation and access capabilities of that model data. Thus, model splicing provides selective access to model data without exposing the entire DE model file, controlling access to the encapsulated model data based on user access permissions. Model splicing also provides DE models with a common, externally accessible application programming interface (API) for programmatic execution of the DE model. Splices of models thus generated can be shared, executed, modified, or further spliced ​​independently of the native DE tools and development platforms used to generate the input digital model. Standardization of DE model data and generalization of API interfaces and functions make DE model type files accessible outside of native software environments, enabling linking of different DE model type files that were previously incompatible. Model splicing further enables the scripting and coding of DE operations, including different DE tools within a standard corpus of program code, facilitating the generation and training of artificial intelligence (AI) and machine learning (ML) models, and manipulating DE models at different stages and workflows of the DE process or lifecycle.

[0186] Digital threads are created through user-initiated and / or autonomous linking of model splices. Digital threads are intended to connect two or more DE models, enabling traceability throughout the entire systems engineering lifecycle and facilitating collaboration and sharing among individuals performing DE tasks. In digital threads, appropriate outputs from previous digital models are provided as inputs to subsequent digital models, enabling a flow of information. In essence, digital threads can be viewed as a communication framework or data-driven architecture that connects traditionally siloed elements and enables the flow of information between digital models. The extensibility of model splicing to diverse DE models and tools allows for the scaling and generalization of digital threads to represent each stage of the DE lifecycle.

[0187] A digital twin is a real-time virtual replica of a physical object or system, with a bidirectional information flow between the virtual and physical domains, enabling monitoring, analysis, and optimization. Model splicing allows individual DE model files to be created as executable splices and linked autonomously and securely, making it possible to manage numerous DE models as a unified digital thread. This capability extends to connecting previously ininteroperable DE models to create a digital thread, receiving external performance and sensor data streams (e.g., aggregated data from DE models or linked data from physical sensors), calibrating the digital twin with data streams from physical sensors outside the native digital twin environment, and receiving expert feedback to enable simulation and parameter optimization.

[0188] Unlike a digital twin, a virtual replica, or simulation, is a mathematical model that mimics real-world behavior to predict outcomes and validate strategies. Digital twins use real-time data for two-way communication, while simulations focus on scenario analysis and outcome prediction. In other words, a digital twin reflects the state of a physical system in time and space. A simulation is a set of operations that reflects possible future states and outcomes that a digital model may develop in the future. A simulation model is a DE model in the context of the IDEP disclosed herein.

[0189] When testing different designs, such as variations in wing length or chord dimensions, multiple digital twins (sometimes in increments of 100 to 1,000 seconds) are created, bridging the gap between design specifications and system examples, enabling seamless updates and tracking of changes across a vast number of variables. This is detailed in the context of the diagram. 1. For example, if you create a system with three variations, there will be a digital twin with specific measurements for each. These digital twins are accessible and updatable via API function scripts, allowing for easy input of new measurements from physical parts during the manufacturing process. By autonomously linking with the appropriate data, the digital twins are updated to reflect the actual measurements of the parts, maintaining traceability and ensuring accurate data representation across hundreds or thousands of models.

[0190] Exemplary model splicing setup Fig. 7 is a schematic diagram illustrating an exemplary model splicing device based on several examples of the present invention. In particular, Fig. 7 is a schematic diagram illustrating an example of splicing an embedded CAD model.

[0191] In this disclosure, a “model splice,” “model wrapper,” or “model graft” of a particular DE model file includes (1) a locator or copy to a digital artifact (including model metadata) extracted or derived from the DE model file, and (2) a locator or copy to a splice function (e.g., an API function script) applicable to the DE model data. A model splice may take the form of a digital file or multiple digital files. A locator refers to a link, address, pointer, index, access key, unified resource locator (URL), or similar reference related to the aforementioned DE digital artifact or splice function, which may be stored in an access-controlled database, a cloud-based storage bucket, or other type of secure storage environment. The splice function provides a unified and standardized input / output API or SDK endpoint for accessing and manipulating DE model data. DE model data differs by model type, and model splices are associated with model type-specific input / output schemas. Depending on a particular user application or data access restrictions, one or more different model splices may be generated from the same input DE model file. In some contexts, the shorter terms "splice," "wrapper," and "graft" are used to refer to spliced, wrapped, or grafted models.

[0192] Model splicing is the process of generating model splices from DE model files. Similarly, a model splicer is program code or an uncompiled script that performs model splicing on a DE model. A DE model splicer for a particular DE model type, when applied to a specific DE model file of that DE model type, retrieves, extracts, or derives DE model data associated with the DE model file, generates and encapsulates splice functions, and instantiates API or SDK endpoints into the DE model based on the input / output schema. In some implementations, a model splicer consists of a collection of API function scripts that can be used as templates for generating DE model splices. "Model splicer generation" refers to the process of setting up a model splicer, establishing a comprehensive framework or template from which individual model splices are derived.

[0193] Therefore, a DE model type-specific model splicer extracts or derives model data from a DE model file, or stores that model data in a model type-specific data structure. The DE model splicer further generates or enumerates splice functions that call native DE tools or API functions for application to the DE model data. A DE model splice for a specific user application contains or wraps user application-specific DE model data and splice functions, accessing only limited portions of the original DE model file, and enabling collaboration and sharing with stakeholders in that user application.

[0194] Furthermore, a document splicer is a type of DE model splicer specific to document models. A "document" is an electronic file that provides information as an official record. Documents include human-readable files that can be read without specialized software, and machine-readable documents that can be viewed and manipulated by humans using specialized software such as word processors and web services. Therefore, documents contain natural language-based text and graphics that can be read directly by humans without requiring additional machine compilation, rendering, visualization, or interpretation. For a specific user application, "document splices," "document model splices," and "document wrappers" can be generated by wrapping user application-specific document data and splicing functions (e.g., API function scripts). This allows text to be revealed at the component or sub-component level (e.g., title, table of contents, chapter, section, paragraph) through APIs and SDK endpoints. It also allows access to and modification of parts of the original document or document template, enabling collaboration and sharing with stakeholders of the specific user application while minimizing manual referencing and human error.

[0195] In the splicing example of a CAD model shown in the figure, the CAD model file diesel-engine.prt704 proceeds through the model splicing process, consisting of 7 data extraction steps 10720 and a splice function generation step 730. This input differential equation model 704 is provided in a file format native to the specific DE tool (.prt). Data extraction is performed through a DE model crawl agent implemented as a model crawl script within the model splicer, which crawls the input DE model file and extracts model data with metadata 722. Metadata is data that can be viewed without opening the entire input DE model file and may include entries such as file name, file size, file version, last modified date and time, and potential user input options identified from user input 706. Model data is extracted or derived from the input DE model and includes, but is not limited to, parts (e.g., propeller, engine cylinder, engine cap, engine radiator, etc.), solids, surfaces, polygon representations, materials, etc. Once the model splicer crawls the model file, it determines how the model data is organized and accessed. This is fundamentally defined by the DE tool 702, which is used for splicing DE models and establishes the model data schema. This data schema describes the structure and format of the model data, some of which are transformed or used to create I / O API endpoints with corresponding I / O schemas. In some implementations, the model includes metadata 722 which may be stored in access-restricted storage 726, for example, "customer buckets" in 3 customer environments, as shown in Figure 10.3, so the model is spliced ​​as follows 742, 744, 7 input DE models can be generated on demand 46 704.

[0196] The model splicer further generates splice functions (e.g., API function scripts) from native APIs associated with the input CAD model. In this disclosure, “native” and “primal” refer to existing DE model files, functions, and API libraries associated with specific third-party DE tools. These tools include both proprietary and open-source options. Native APIs may be provided by proprietary or open-source DE tools. For example, the model splicer may generate an API function script that calls the native API of a native DE tool to perform functions such as: Hide Parts (parts_list), Generate 2DView(), etc. These model type-specific splice functions may be stored in a splice function database for on-demand generation of individual model splices. The platform API consists of a catalog and specifications of splice functions provided by different model splices supported by IDEP, and orchestration scripts that link multiple model splices. This platform API is a common, universal, and externally accessible platform interface that hides native APIs.7 This allows engineers from different disciplines to interact with unfamiliar DE tools, and enables free interoperability between DE tools that were previously incompatible.

[0197] Next, based on user input and desired user applications, one or more model splices or wrappers are generated. These wrap some or all of the model data required by the user application in splice functions or API function scripts, which are then applied to the original input model or the wrapped model data to perform the desired operations and complete the task requested by the user. In various implementations, model splices may take the form of a digital file or multiple digital files, and model splices may combine or replace the aforementioned DE digital artifacts or locators or copies of splice functions. By combining selective portions of the model data, any number of model splices / wrappers can be generated. API function scripts provide a unified and standardized input / output API endpoint for accessing and manipulating DE models and DE model data. These API handles and endpoints can be used to execute model splices and establish links with other model splices without directly calling native APIs. Such API endpoints are formatted according to the input / output scheme of the DE model files and DE tools used, and may be accessed by orchestration scripts or platform applications that act on multiple DE models.

[0198] In some implementations, API function scripts input or output to DE models or DE model splices at runtime. "Input" splice functions or "input nodes" (733) are model modification scripts that enable updates or modifications to the input DE model. For example, updating a model might involve changing model parameters or configurations through input splice functions. "Output" splice functions or "output nodes" (734) are data / artifact extraction scripts that enable data extraction or derivation through the DE model's model splice. API function scripts may invoke native API function calls of native DE tools. Artifacts are the results of execution by output API function scripts within a model splice. Multiple artifacts can be generated from a single DE model or DE model splice. Artifacts may be stored in restricted-access cloud storage (726) or other similar restricted-access customer buckets.

[0199] One of the advantages of model splicing is that it provides minimal privileged access control capabilities in the zero-trust implementation disclosed herein, in the various deployment scenarios discussed with reference to the diagrams, 4, and in the context of the IDEP implementation architecture discussed with reference to the diagrams. 3, the original DE input model 704 and model data storage 726 may reside in customer buckets in the customer environment. 3. Splice functions make native API calls to the database 702. 702. Execution or calling of splice functions 732 may rely on job-specific authentication or authorization by the DE tool's own license (e.g., if present in the customer environment) 3, 103 and / or the applicant's information security clearance level. Thus, model splicing unbundles monolithic access to digital model type files as a whole file and instead provides specific access to a subset of functions that enable limited, purposeful, and auditable operations on a subset of model type files composed of component parts or assembled atomic units.

[0200] Digital sliding of DE models using model splicing Fig. 8. A schematic diagram illustrating digital threading of DE models by model splicing, based on several examples of the present invention. Digital threading is intended to connect two or more DE models, enabling traceability throughout the entire system engineering lifecycle and allowing collaboration and sharing between individuals performing DE tasks.

[0201] Linking model splices generally refers to jointly accessing two or more DE model splices through API endpoints or splice functions. For example, data may be retrieved from one splice to update another (e.g., the input splice function of the first model splice calls the output splice function of the second model splice). Data may be retrieved from both splices to generate new output (e.g., output splice functions are called from both model splices); or data from a third splice may be used to update both the first and second splices (e.g., input splice functions are called from both model splices). In this disclosure, "model linking" and "model splice linking" are used synonymously, and linked model splices are mapped to corresponding linked DE models. Similarly, linking DE tools refers to jointly accessing two or more DE tools through a model splice, where model splice functions encapsulating different DE tool functions interoperate and call them, or where orchestration scripts jointly execute DE tasks.

[0202] Therefore, model splicing enables a mechanism to convert individual digital model files into model splices, allowing them to be linked autonomously and securely, and enabling the management of numerous digital models as a unified digital thread written in scripts. Within the IDEP described here, a digital thread is a platform script that invokes platform APIs and facilitates, manages, or orchestrates workflows through linked model splices. Model splice links provide a communication framework or data-driven architecture that connects traditionally siloed elements and enables the flow of information between digital models via corresponding model splices. The extensibility of model splicing across diverse types of digital models allows for the scaling and generalization of digital threads that represent each stage of the DE lifecycle and instantiate or update digital twins as needed.

[0203] In the specific example shown in the diagram, the orchestration script 894 is written in Python code and is designed to interact via the following API endpoints. 892 was for determining whether the CAD model met the total mass requirements. API endpoint 892 is an output splice function and is part of the platform API 890. The platform API 890 includes not only splice functions but also platform scripts and orchestration scripts. 894 is itself.

[0204] Orchestration script 894 is mainly divided into three stages. 1. Retrieving data from CAD model splices: You can perform a computer-aided design (CAD) model splice by sending a POST request through the IDEP platform API871. This model splice provides a unified interface for modifying and retrieving information about the CAD model881. Parameters of the CAD model, such as hole diameter, notch opening, and flange thickness, are sent in the request and can be set in the input splice function. The total mass of the CAD model is derived from the model parameters and can be retrieved in the output splice function. The response from the platform API includes the total mass of the CAD model881, and a unified resource identifier / locator (URL) for the CAD model is provided. The response may also include a URL for an image of the CAD model. 2. Retrieving data from SysML model splices: You can send another POST request via the IDEP platform API to execute a System Modeling Language (SysML) model splice. SysML is a general-purpose modeling language used in systems engineering. The output function for model splices retrieves the total mass requirements of the system from the SysML model. The response from the platform API contains the total mass requirements of the system. 3. Align the variables and verify that the requirements are met: The total mass 881 from the CAD model is compared to the total mass requirement 882 from the SysML model. If the two values ​​are equal, a message will be printed indicating that the CAD model meets the requirements. Otherwise, a message will be printed indicating that the CAD model does not meet the requirements.

[0205] In short, the orchestration script 894 can be implemented in the application plane as shown in Figure 601 of IDEP 00.1, by linking the digital model 881 and 82 via the Model Splice API call. The orchestration script 894 is a scripted platform application that modifies a CAD model, obtains the total mass of the modified CAD model, obtains the total mass requirements from the SysML model, and compares both values ​​to verify that the CAD model meets the requirements. Some implementations utilize a set of functions within IDEP 100 that allow the platform application to act on multiple DE models.

[0206] Model Splice Plane Fig. 9 This is a schematic diagram illustrating how to connect DE model splices on a splice plane, comparing the presence or absence of model splicing with digital splicing, based on several examples of the present invention. The lower model plane 180 shows current digital threading technology, where each small ellipse represents a DE model and shows a link between any two DE models. Numbers 982 and 984 require connection to the central home 910, and the possibility of additional connections from each model to all other models. The central home 910 consists of program code that can interpret and manipulate original DE models of different model types. For example, under the management of platform 9 experts, data can be prepared from the digital model, divided into 82 formats accessible by the digital model, and modifications to the digital model are possible via the 84984 native API, which will be propagated to the digital model. Feedback from the digital model will require similar processing via the platform. Data from the digital model is converted to a format accessible by the digital model via the 82982 native API. This hub-and-spoke architecture cannot scale to hundreds or thousands of digital models involved in typical large-scale DE projects because model updates and feedback are only possible through the central platform.

[0207] In contrast, when a DE model is spliced, each original model is represented by a model splice, which is represented by associated model data and integrated, standardized input / output API endpoints (shown in the upper splice plane) 170. Splices 170 within the splice plane are connected via scripts (e.g., Python scripts) that call API endpoints or API function scripts, and may also follow a DAG architecture as illustrated with reference to Figure 1 and Fig. 6. Note within Figure 1, only the set of generated splices 170 is shown within the splice plane in Figure 9. Scripts linking the model splices are also shown within the splice plane for illustrative purposes. Such scripts are referred to in this disclosure as orchestration scripts or platform scripts and orchestrate workflows through digital threads based on the interconnected DE model splices. Furthermore, the splice plane 170 is shown in Figure 1. As part of IDEP 100, some implementations use splice planes for illustrative purposes, which are implemented behind the customer's firewall and may also be part of the DE platform agent, as illustrated in the various deployment scenarios shown in the diagram.4. In other words, the individual API function scripts generated by model splicing by the DE platform agent may be customized to call proprietary tools that the customer has access to in their private environment.There is no centralized platform9 that has proprietary access to all native tools associated with all the individual digital models shown in the diagram10.9 that is necessary. Instead, the orchestration script calls a universal API function script that may have different implementations in different customer environments.

[0208] Therefore, model splicing enables model splicing like model splicing from 9 digital models (72982 and 74984) to intentionally and directly access each other's data, enabling the creation of a model-based "digital mesh" that leverages platform scripting (44) and allows for autonomous linking without expert input.

[0209] An additional advantage of moving from model airplanes to spliceplanes is that the DE platform allows for the creation of multiple splices for each native model (see diagram, e.g.). Each has a different subset of model data and a customized API endpoint for the purpose of the splice. For example, model splices are used to generate multiple digital twins (digital twins) that map the design of a physical product, process, or object into a virtual space. Bidirectional data exchange between the physical object and its digital object twin enables testing, optimization, verification, and validation of the physical object in the virtual world. By selecting the optimal combination of digital model configurations and architectures from parallel digital twins based on model splices, each may respond differently to the same feedback from the physical object.

[0210] Supported by model splicing, digital threading, and digital twin capabilities, the IDEP described in this book connects DE models and DE tools, enabling simple and secure collaboration of digital engineering data across model sources such as engineering fields, tool vendors, networks, government agencies and organizations, special program offices, contractors, SMEs, and Federally Funded Research and Development Centers (FFRDCs), as well as University-Affiliated Research Centers (UARCs). Application example 9 of IDEP 50 is shown on the right side of the diagram. It demonstrates how to integrate data from diverse organizations and enable cross-domain collaboration while maintaining data security, traceability, and auditability. Here, DE models from multiple vendors and component constructors are spliced ​​or wrapped by the IDEP agent, and data artifacts are extracted with data protection. Converting DE models into data artifacts enables cross-domain data transfer and protects critical information. This allows model owners to have complete control over their DE models using their existing security and IT stacks, continue using the DE tools best suited to their purpose, and maintain the modeling schema / ontology / profile that best suits their purpose. IDEP transforms the DE model into microservices, providing the least privileged data bits to reach relevant stakeholders without the DE model leaving its home server or being replicated or delegated. IDEP also offers simple data access and digital threading options through secure web applications and secure APIs.

[0211] DAG representation of thread tasks Model splicing provides a unified interface between DE models, allowing model and system updates to be represented in interconnected and pipelined DE tasks. Figure 10 shows an example of a directed acyclic graph (DAG) representation. According to some embodiments of the present invention, 10 pipelined DE tasks related to digital threads are shown. In the figure 10, tasks (e.g., 894) executed through a digital thread orchestration script are structured as nodes in the DAG. Thus, actions are interconnected and executed in a pipeline connecting splices of DE models with corresponding ranges of parameter values. Therefore, digital threads can be created by constructing appropriate connections at the relevant endpoints between each model splice of the corresponding model through an interpretable DE platform script.

[0212] This refers to a fig. A DAG with 1 to 8 thread processing tasks is constructed from digital threads and is part of the application plane of the DE platform.160 Different DAGs may target different DE actions. For example, building or updating a digital twin in a fig, or building or updating a virtual environment, each have their own DAG.124 Model splicing transforms DE models into data structures accessible via APIs, allowing DE actions to be performed using software development tools, from simple Python scripts to complex DAGs. Digital threads in model splicing eliminate the scalability issues of digital thread management and accelerate the digital design process, including design updates based on external feedback.

[0213] Following the above description of the basic and core elements of IDMP / IDEP, we will now describe in detail the system and methods of interaction with live digital objects.

[0214] Multimodal interface with digital model files Figure 11 is an exemplary system diagram illustrating the process of interacting with live digital objects in an interconnected digital model platform (IDMP) according to several examples of the present invention. In particular, Figure 11 provides a schematic representation of the system that enables user interaction. Figure 1102 has various multimodal interface modules and data. Figure 1104 is a multimodal interface on the IDMP according to several examples of the present invention. In Figure 11, the multimodal interface 1104 allows the user to access live digital objects. Figure 1102 is the key to access live digital objects. Figure 1102 is the acquisition of 50A through IDMP application. 1122, live digital objects 1150A may include digital artifacts.

[0215] The system may include at least one hardware processor responsible for executing program code, 1211 for implementing modules, 1411 (details below). The system may include access to at least one non-temporary physical storage medium accessible by at least one hardware processor, 1112 for storing program code executable by hardware processors. The program code may be stored and distributed across two or more non-temporary physical storage media and executed by two or more processors.

[0216] The system may include a multimodal interface for receiving input from 11 users 2002; the embodiment of a figure; 11 users 1102 equipped with a virtual reality (VR) or augmented reality (AR) headset 1106. In practice, the multimodal interface may include AR / VR headsets, interactive gloves, cameras, and even spatial or conversational peripherals that allow users to interact with data provided by the IDMP application 1122.

[0217] The system may include an API interface 1138, which represents a common externally accessible application programming interface (API) through which digital artifacts pass 1140A, which is extracted from the model splice 1134. The model splice 1134 (or more generally, the model representation) may be generated from a model file 1132 using a model splicer 301132. The model splice 1134 can be configured to selectively access model data within the model file 1130, for example, digital artifacts 1140A.

[0218] The system may include an access control mechanism 1136 which may be part of the model splice 1134. The access control mechanism 1136 may provide access to the recovered digital artifacts 1140A which is based on user access rights 1102 (e.g., based on the user's security level) to ensure secure and controlled data retrieval.

[0219] The IDMP application 1122 can generate and maintain live digital objects 1150A, and access to recovered digital artifacts 1140B is available. In some implementations, the IDMP application 1122 also acts as an update engine that can update live digital objects 1150A, based on input from one or more users or software agents. This also includes reflecting changes made to the digital artifacts 1140A and inputting them into the live digital object 1150A. The IDMP application 1122 enables the techniques described here by orchestrating user interactions through a multimodal interface 1104, which may also allow access to the digital artifacts 1140A and inputting them into the live digital object 1150A.

[0220] As an example, an IDMP application 1122 can receive live digital objects 1150A, which may include digital artifacts 11B extracted from model files 11 model representations (e.g., model partitions) 301134) including locators for model type-specific digital model data and metadata. The IDMP application 1122 can initiate a connection to a multimodal interface 1104 configured to receive input (and output) from at least two different modalities, including conversational modality and spatial modality.

[0221] The IDMP application 1122 can receive the user's security level 1102 and, based on the security level, decide whether to grant access to (i.e., "permission to access") and / or modify (i.e., "permission to modify") the digital artifact.

[0222] IDMP application 1122 may also determine the accessible portion of a live digital object based on the user's security level. In the figure, the accessible portion of a live digital object is shown with a solid line, while the remaining (inaccessible) portion is shown with a dotted line. IDMP application 1122 outputs a digital artifact with user access permissions. In the figure, the digital artifact is shown to be accessible and is part of the accessible portion of the live digital object.

[0223] IDMP application 1122 may also receive conversational inputs and spatial inputs related to digital artifacts from the user via a multimodal interface 1140°C. Based on the user's permission to modify digital artifacts 1140C, IDMP application 1122 may also generate modified digital artifacts from digital artifacts 1140A by digital model representation (e.g., model splice) 1134).

[0224] Fig. 11 Thus, it shows the flow of data and the interaction between the multimodal interface module and the data 1120, 02 inputs 1104 to the generation of modified digital artifacts and / or live digital objects through the multimodal interface 11 starting from the user.

[0225] Figure 12 illustrates an example workflow showing how different user interfaces enable specific user operations within a digital engineering platform, in accordance with the exemplifications of the present invention. Figure 12 displays user interface options, including both conversational and spatial computing interfaces, each offering different functionalities, for various aspects which will be discussed in more detail below. In a sense, module 1202 shows a conversational interface which may further include IVR or chatbot submodules. In one implementation, a user can upload a digital engineering (DE) module and utilize the conversational interface which we used to iterate on a specific use case. In another example, module 1212 includes a spatial computing interface which further includes direct interaction, context input, or a shared haptic mechanism. In one implementation, a user can create a new DE model or manage a digital twin simulation relative to the physical world. As further shown in the figure, a user can interact with the input-side platform and also interact with the output-side platform.

[0226] Digital Engineering through AI-Assisted Script Generation In various implementations, a method has been proposed to convert scripts within IDEP into embeddings and generate scripts that train one or more transformers to perform DE tasks. Customer data sovereignty considerations are discussed in detail in PCT application number PCT / US24 / 38878 (case number IST-03.002PCT).

[0227] Most scripts used in IDEP fall into one of the following two categories:

[0228] 1. The API script manipulates splicing on the splicing plane of the model (see diagram). 1). It utilizes APIs of specific digital engineering tools (e.g., CAD, CFD, FEA).

[0229] 2. Orchestration scripts that manipulate digital threads or digital twins on the application plane or control / analysis plane (see figure). 1) These can call API scripts through microservices (see PCT application numbers PCT / US24 / 18278 (case number IST-02.001PCT) and PCT / US24 / 27898 (case number IST-03.001PCT)) or DAG tasks (see figure) to coordinate multiple different DE tools.

[0230] Fig. 13. In accordance with one embodiment of the present invention, we demonstrate a generalized AI-assisted design process on a digital engineering platform. As an embodiment of Fig. 13, the three main components used in AI-assisted digital design are as follows:

[0231] 3. Contextual AI Model (1304): IDEP receives access to a contextual AI model (1304) and input prompts (1302). Input prompts (1302) are typically prompts from IDEP users (e.g., human users or software agents). The contextual AI model may be based on one or more large transformers or LLMs (e.g., the contextual AI model). (1304) may be a closed-source LLM such as GPT4. Its role is related to steps and associated subsystems (1306) of the DE task. It may receive user input DE prompts (1302) and generate a list of steps and subsystems required for the selected syntactic AI model to fulfill the DE prompt (1306), with the selection of one or more trained syntactic AI models. In one implementation, the contextual AI identifies the steps and subsystems that perform the workflow task related to the prompt, and the user can select the corresponding syntactic AI model.

[0232] In various forms, a DE prompt (1302) is a user request to perform a DE task that involves accessing a model splice. Examples of DE prompts include: a. General user prompt: "Create a gear with 20 teeth, a pitch diameter of 50 mm, and a maximum torque of 20 Nm or less, with a lifespan of 3 years." b. Lower-level user prompt: "Use open-source tools to perform static and dynamic analysis of a 3D CAD model of a spur gear according to the provided dimensions. Evaluate the gear behavior of three different materials (e.g., sintered iron, injection-molded nylon, 3D-printed ABS)."

[0233] 4. Syntax AI Model (1308): The trained syntactic AI model is selected by the user based on suggestions from the context AI model, or it may be selected by the context AI model. It receives a list of steps or subsystems required to satisfy DE prompts (1306) and generates one or more scripts (1310) to implement the DE steps that satisfy the DE prompts, where these scripts contain variables for the parameters to be substituted. The syntactic AI model is based on open-source transformers or LLMs and may be trained to generate API scripts or orchestration scripts. Thus, in one example, the trained syntactic AI model generates template scripts containing API or splicing scripts, where the template scripts contain variables (i.e., placeholders for parameters related to the digital task). The generation of variable-parameter scripts (i.e., template scripts) enables the anonymization of trade-sensitive parameters using a variable number of "parameter placeholders," a process known as "placeholder anonymization." This process ensures sovereignty over customer data, as described below. The script database (1326) may be provided by IDEP for training, fine-tuning, or providing runtime context information to the syntactic AI model (described below). The syntactic AI model can also be trained (for 13 platform API documentation, spatial and conversational API documentation for peripherals (e.g., Microsoft HoloLens, Apple Vision Pro, etc.), and multimodal interface API documentation).

[0234] 5. Parameter substitution process (1312): The parameter substitution process replaces the variables identified in the syntactic AI model (1310) with enterprise sensitive parameters (1314). In some implementations, the received script may be a template script containing placeholder variables. The parameter substitution process (1312) generates a parameter substitution script (e.g., an orchestration script) to implement the design steps related to the DE prompt (1314). Placeholder variables in the script are parameters (1314). Enterprise sensitive parameters are typically derived from enterprise documents (1328) and may look like this: a. Inserted by the user, or selected by the user from a list extracted from corporate documents. b. A user-selected item from a list generated by the Enterprise AI module from enterprise documents. c. Inserted by the algorithm from the parameter table, or d. It is inserted by the Enterprise AI module.

[0235] In some implementations, the parameter substitution process maps variables to corresponding software tool documentation within the customer environment. Software tool documentation may include operational manuals, programming and scripting functions, feature lists / manuals, APIs, specification files, requirements files, certification files, enterprise documentation, or a combination of these. In some practices, the parameter substitution process maintains and periodically updates a variable mapping table, which shows a table of variables in the customer environment and their corresponding (i.e., mapped) software tool documentation. In one example, to determine the value of a placeholder variable, the parameter substitution process might look up that value in the mapped software tool documentation. In another example where the parameter substitution process uses a substitution machine learning (ML) model, variable-document pairs in the variable mapping table can be used to train the substitution ML model.

[0236] One example is the parameter substitution process, which uses a substitution machine learning (ML) model as shown here (1312). In one example, the characters (1310) generated by the syntactic AI model may not contain variables (i.e., placeholders for parameter values) and may therefore be output as a parameter substitution script (1312) without going through the parameter substitution process (14). Once the scripts are ready (1314), they can be executed (1316) and the resulting designs are output (1318) for migration to IDEP or the customer environment.

[0237] In some implementations, syntactic AI models and substitution ML models can be trained using either a Search-Augmented Generative (RAG) or Low-Rank Adaptive (LoRA) method. The RAG-based approach leverages a knowledge base of code examples, documentation examples, and platform APIs. The RAG-based approach enhances the generative capabilities of syntactic AI models (and / or substitution ML models) by supplementing them with contextually relevant information for the requested digital task, improving accuracy and detail. Technically, RAG includes a search mechanism to retrieve relevant documents to aid in the generative process during inference, making it suitable for tasks requiring a broad knowledge base. The Search-Augmented Generative (RAG) framework and methodology are described in more detail below: Lewis et al., "Search-Augmented Generative for Knowledge-Intensive NLP Tasks," arXiv:2005.11401, 2020. The full text is embedded here by reference.

[0238] In contrast, the LoRA approach fine-tunes syntactic AI models (and / or permutation ML models) for specific digital tasks and workflows, introducing low-rank updates to maintain efficiency while significantly reducing computational and memory requirements. LoRA works by adding a low-rank decomposition matrix to the model's existing weights, rather than directly altering the original weights. This approach enables task-specific adaptation with minimal additional parameters, reducing the computational resources required for fine-tuning and allowing for rapid adaptation to new tasks and domains without the need to retrain the entire model.

[0239] LoRA fine-tuning is task-specific, using smaller datasets than the base model's original training dataset, and including examples representative of the target digital task. The data must be carefully curated to avoid bias and errors. Therefore, syntactic AI models can be fine-tuned using datasets containing sample context data and template script pairs specific to digital tasks such as generating budget audit reports or digital engineering certification reports. Similarly, alternative ML models can be fine-tuned using datasets containing sample template and orchestration script pairs specific to digital tasks. In practice, LoRA fine-tuning datasets include sample template scripts, orchestration scripts, platform APIs, software tool documentation, and enterprise documentation. The Low-Rank Adaptation (LoRA) technique is described in more detail below: Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685, 2021. The full text is embedded here by reference.

[0240] LoRA is a fine-tuning technique for more efficiently updating and specializing syntactic AI models, while RAG aims to extend the effective knowledge of syntactic AI models by providing contextual information at runtime. Therefore, LoRA is particularly advantageous in resource-constrained environments and highly specialized workflows, as it allows models to be adapted with fewer parameters. RAG excels in scenarios requiring broad and detailed information retrieval, while LoRA is ideal for efficiently generating orchestration scripts for specific digital processes. Some implementations include RAG-based LLM agents or LoRA adapters with LLM agents for syntactic AI model manipulation, and can be customized for specific sets of digital tasks.

[0241] In some implementations, the parameter substitution process (1312) may include the generation of scripts with dummy parameters (1310), which are replaced with enterprise-sensitive parameters as described below. Various options for implementing the parameter substitution process are disclosed in PCT application number PCT / US24 / 38878 (case number IST-03.002PCT).

[0242] The following are example parameters in the context of API and orchestration scripts. By replacing the variables with parameters, the generated script can be executed on IDEP and fulfill the user's input DE prompts.

[0243] User Feedback Within the AI-assisted script generation pipeline, multiple user input and feedback modalities can be implemented. This illustrates the embodiment of a fig. It shows 13 or fewer user interactions.

[0244] Step 1302 can start with a simple task request like, "Design a 20-tooth gear..." or "I want to make a plastic chair..."

[0245] In step 1320, the user interacts with a reinforcement learning human feedback (RLHF) loop, allowing them to approve or reject workflow steps and tools within the described digital thread. For example, the user can determine recommended material selection models or simulation models.

[0246] In step 1322, the user can thoroughly review the digital threads provided by the system. The user reviews the proposed steps and their parameters. The user can update them as needed. Modifications include changes to the model and DE tools, their respective parameters, and updates to these parameter values. For example, the user can adapt software tools and machining parameters, and provide updates to the finite element analysis (FEA) model.

[0247] In step 1324, within the RLHF loop, the user can review, select, or reject the proposed algorithm script. For example, the user can analyze coding algorithms or machine learning models and decide whether to accept or reject them.

[0248] Fig. This shows the data flow through 13 platforms, and in addition to the user interactions shown in this flow, user input options to the platforms are added over multiple iterations. In some implementations, the user may go through the following interactive steps: 1. When uploading a model, 2. Selecting functions on the digital engineering platform (application plane or splicing plane) - see Figure 1). 3. Functional changes to the digital engineering platform (on the application plane - see diagram). 1), and 4. Implicitly assess whether the digital model meets the certification requirements (e.g., insert the digital model into the certification document and make no further changes). This implicit assessment is an important user feedback element.

[0249] Type of character When generating API scripts, the main building blocks may have the following specific characteristics: 1. Contextual AI Models (1304): Contextual AI models generate the purpose of the code to be generated, the tools to interact with, and broader engineering challenges belonging to areas such as aircraft design. This data defines the framework of the task and provides a high level of understanding of the steps to be taken. This contextual data is usually difficult to infer purely from DE prompts alone. Therefore, transformers such as Large-Scale Language Models (LLMs) are particularly effective because they can process large amounts of data and extract high-level themes and concepts. 2. Syntactic AI Models (1308): The transformers and LLMs that make up a syntactic AI model are trained to generate actual code that interfaces with the APIs of a specific DE tool. Because these transformers and LLMs are trained on datasets of similar API interactions, they can capture the subtle nuances of how these APIs work. While API scripts are by definition tool-specific, syntactic AI models trained to generate API scripts may be trained on the native APIs of multiple specific tools. Tool specialization is possible, but generally, syntactic AI models are trained on API scripts within a digital engineering platform, mixed with native APIs and platform APIs that implement business logic and associated orchestration scripts. 3. Parameter Substitution Process (1312): The parameters in the API script are highly specific to each use case and include part dimensions (e.g., aircraft wings), fluid viscosity (e.g., for CFD simulations), or material properties of FEA models. Because these are deterministic and customer-specific, they may be provided directly by the user or in a highly reliable manner, and should be incorporated into the code in a deterministic and consistent manner (e.g., through the template system specified in the PCT application number PCT / US24 / 35885 (case number) IST-02.002PCT)).

[0250] When generating microservices or DAG task scripts, the main building blocks may have the following specific characteristics: 1. Contextual AI Model: The contextual AI model provides a workflow to execute the target DE task and identifies the necessary DE tools. 2. Syntax AI Models: Transformers may generate code that calls necessary API scripts and ensure correct order, dependencies, error handling, etc., throughout the manipulated DE model files. These scripts can also be seen as orchestrating the workflow of the entire engineering task. 3. Parameter Substitution Process: Parameters here include the specific order of tasks, required waiting periods between tasks, and how to handle outputs and error messages. These may be provided directly and deterministically inserted into the script, or they may be inserted by the Enterprise AI module.

[0251] Figure 13 illustrates the pipeline for generating a script to execute a DE task. Thus, a cascaded contextual and syntactic AI model generates program code, and a parameter substitution process ensures that parameters are inserted. This mechanism allows for the flexible generation of highly complex and tool-specific code while maintaining an appropriate level of control and specificity.

[0252] Parameter substitution as a zero-knowledge indicator In one example, IDMP's zero-knowledge (ZK) architecture is implemented to prevent customer data deemed sensitive by the IDMP Software Development Kit (SDK) from being transmitted via the IDMP API. The purpose of this ZK is achieved through a cryptographic tokenization process. Cryptographic tokenization identifies sensitive data through customer input and maps each sensitive data element (e.g., digital model, digital artifact, document) with a cryptographic token and cryptographic identifier. Each cryptographic token contains metadata that describes the data element. In cryptographic tokenization, the metadata of the cryptographic token, rather than the data element itself, is used to train the syntactic AI model. Therefore, the training dataset for the syntactic AI model may include a customer data sovereignty training dataset consisting of sample context data related to a sample digital task and corresponding sample template scripts. The generation of each sample template script involves receiving an orchestration script, implementing the associated digital task, identifying sensitive data elements within the orchestration script, and replacing each sensitive data element with the mapped metadata.

[0253] Cryptographic tokenization is a cryptographic tokenization (detokenization) process in which, when data elements are used outside the customer environment, sensitive data is replaced with a cryptographic identifier, and the mapped data element is exchanged for a cryptographic token, and then the cryptographic token is exchanged for use within the customer environment. Therefore, the ZK architecture stores sensitive data elements in the customer environment (e.g., on the customer's network).

[0254] Parameter substitution is another element of the ZK architecture. Specifically, the parameter substitution process contributes to the ZK architecture by mapping common parameter names and details of generic API functions (e.g., function names, inputs, outputs) to specific software tool resources and software tool functions within the customer environment. Therefore, the orchestration scripts generated by the syntactic AI model support the ZK architecture by requiring explicit parameter substitution steps within the customer environment.

[0255] Multimodal access to live digital objects Integrating VR / AR technology into live documents, live boards, and live spaces presents several challenges that need to be addressed to enhance user interaction, collaboration, and efficiency. These challenges can be broadly categorized into four areas. 1. Navigation and Discovery: Due to the vast amount of data artifacts and their complex connections, users face difficulties navigating and finding the correct elements within live documents and live boards. This complexity makes it difficult for users to quickly find the information they need. Traditional two-dimensional directory-style lists and parent-child relationship graph layouts provide users with a starting point to find all potential data artifacts and their possible versions, and then initiate a sorting process to narrow it down to the most relevant ones. 2. Navigating and finding data artifacts in a sorting 2D browser or screen interface is already difficult. Efficiently organizing various data artifacts in such a 2D environment presents an additional challenge to avoid overwhelming the user, but an IDMP platform can provide users with additional details. When a user selects a single data artifact, that artifact may already be part of multiple digital threads with zero-trust access rights. The user then selects the version of the artifact that best suits their decision. When necessary and useful additional context is presented, the user needs to quickly filter and prioritize the relevant information, which can easily become cumbersome without an effective system. 3. Dynamically and securely displaying the correct data artifacts within a 3D display environment, based on the user's zero-trust security permissions and available dynamic digital thread contexts, is complex. The display must update in real time and respect user permissions to prevent unauthorized access to sensitive information. 4. Real-time Collaboration: Enabling multiple users to collaborate in real time within a live space (a virtual environment that extends the context of various digital models alongside the physical environment) presents challenges in ensuring the synchronization of changes and respect for permissions. This requires robust tools to manage simultaneous interactions and updates. When multiple users each have unique credentials and collaborate on a single live document or magic dashboard, it becomes more difficult to display the correct data artifacts.

[0256] Exemplary multimodal interface solution 1. Navigation and Discovery: The multimodal interface with IDMP enables the implementation of a fast, dynamic navigation mechanism similar to Rolodex, allowing users to quickly spin and identify the data artifacts they need. While the IDMP enclave orchestrates this process, the multimodal UX / UI navigation provides dynamic interactions and contextual information beyond mere 2D graph navigation. This mechanism enhances navigation using intuitive gestures and visual cues, leveraging the capabilities of current VR / AR and spatial computing interfaces. 2. Sorting: IDMP can implement advanced sorting algorithms using machine learning models trained on user workflows, metadata, and the platform's universal API. These algorithms work in conjunction with visual representations that leverage the multimodal capabilities of various commercially available (COTS) multimodal (AR / VR and spatial computing) devices. This approach classifies and prioritizes data artifacts based on user preferences and past usage patterns. For example, artifacts similar to selected ones appear closer to the user's field of view, while irrelevant ones are placed further away. Furthermore, gesture-based interaction is real-time adapted as the user interacts with the data, enabling rapid sorting and selection. In another example, spatial audio controls assist sorting steps by using directional cues and pitch changes to inform the user of similar or different artifacts. 3. Display: A multimodal interface can create a personalized view for each user and dynamically update the display based on permissions. Secure access control and encryption ensure that sensitive information is visible only to authorized users. Privacy is protected with features to block or prevent unauthorized viewing. Multiple collaborators on the same live document can have their own custom views while being aware that other collaborators can access different artifacts. Users may not know about the existence of certain artifacts unless authorized, or they may view parts of artifacts that are hidden to the extent that they "need to know." See Secure Breakout Rooms and Procedure 1410 and Figure 36 below.14, an example of a personalized display is shown. 4. Real-time Zero-Trust Multimodal Collaboration: IDMP orchestrates digital threads for real-time synchronization among multiple authorized users, ensuring that all changes are instantly reflected in all users' views. Working with a multimodal interface, IDMP supports zero-trust collaboration, maintaining permission-based access control while users can see each other's actions and updates in real time. In some implementations, IDMP performs these operations in a zero-knowledge manner, using tokenized data to create digital threads with relevant artifacts and functions, ensuring that only authorized users can access the data in the environment. Real-time collaboration using live documents and similar dashboards presents several challenges, primarily in securely managing and sharing complex data manipulations without sharing entire files or digital models. Users often need to perform complex calculations, data transformations, and functions and processes specific to digital models. Exposing underlying functions and algorithms to all collaborators can lead to security risks and unauthorized access to sensitive data, which is inconsistent with zero-trust principles. Furthermore, the complexity of these operations can overwhelm users and impair collaboration efficiency. Some implementations utilize Magic functions as black boxes, allowing collaborators to input data artifacts and receive the necessary output data artifacts without revealing the function's internal workings. This abstraction protects sensitive algorithms while enabling full functionality, streamlining collaboration, enhancing security, and maintaining efficiency. 5. AR / VR Interaction in the Digital Thread: IDMP benefits from all user actions and interactions becoming part of a digital thread on the platform. Just as users can place sections of live documents into specific artifacts using text and magic links, or place digital artifacts as dashboards on live boards, IDMP, working with a multimodal interface, can remember locations and user views in virtual or augmented settings. This capability enhances the user experience by maintaining a consistent spatial context by placing virtual objects into specific physical spaces when similar uses are anticipated.

[0257] Navigation and sorting Fig. 14 The flowchart shown here illustrates the navigation and sorting of data artifacts in a multimodal interface, following the example provided. The navigation process for creating a live document, live board, or live space begins with the user initiating a navigation step (14). The user then accesses a repository of new live documents, boards, or spaces (1404) and data artifacts (1406). The system filters these artifacts based on user and network permissions (1408) and classifies them using machine learning models (1410). Authorized artifacts are displayed using dynamic visualization mechanisms such as Rolodex or carousels. 14 The user takes a gesture (1414) and drops it into a live document, board, or space (1416). The user reviews and confirms artifacts, referencing the parent model to ensure contextual clarity (1418). After finalizing the artifacts added to the live document, board, or space (1420), the user saves the document, board, or space (1424). Finally, the system saves the live document, board, or space and grants access based on the initial artifact selection (1424). If the user has already sequenced artifacts, they can complete the live document simply by navigating. The multimodal interface, integrated with IDMP, allows users to manage their cognitive load and provides contextual clarity when identifying the appropriate digital artifacts they are permitted to view and select.

[0258] Real-time zero-trust collaboration is achieved in steps. In 1408, users operating on the same live document can only view artifacts that have been individually permitted. For artifacts that a user is not permitted to access, depending on the implementation, it may be possible to request access or provide a Magic function that provides the required output data without the details of the edited artifact. In some implementations, instead of a user clicking a button, they can request access to an edited function using a gesture. Step 1410 results in the utilization of a machine learning model that classifies artifacts so that a user can move to a subset based on their input and the intended live document. Steps 1414 and 1416 introduce a grab-and-drop mechanism instead of copy-and-paste to improve the user experience and thus improve the user interaction with digital models and digital workflows. Step 1418 enhances the ability to understand the context of a specific data artifact by enabling a user to navigate to a parent digital model and related data artifacts and check if the selected data artifact is suitable for the live document. In some implementations, 14 has a multimodal interface where a user can select and slide a data artifact or display a related parent model with a double tap or double click. Such gesture mechanisms help a user quickly understand the most important contextual information regarding the artifact under review.

[0259] In one example, if a user loses access to or has their permissions changed for a specific artifact, a box indicating that they do not have permission to view the artifact or, in the case of a live space, a cube may be displayed, and custom gestures such as requesting access or knocking on the hidden artifact may be used.

[0260] The sorting process begins after navigation, starting with the user initiating a sort step (1432) and the set of data artifacts selected in the navigation step (1434). The system uses both human expertise and machine learning models to classify these artifacts based on priority, relevance, and dependencies (14). Using sorting tools and dynamic visualization mechanisms, the system displays the priority and order of artifacts by relative size, proximity, and distance (1438). The user uses gestures (1440) to add and drop artifacts into a live document, board, or space in the modified order (1442). The user reviews and organizes the artifacts using additional context such as parent models and dependencies (1444). After finalizing the order of the artifacts (1446), the user places them in a live document, board, or space (1448). Finally, the system saves and approves access to the live document, board, or space containing the sequenced set of artifacts (1450).

[0261] Similarly in the sorting phase, step 1436 leverages the users of the machine learning model to help manage the prioritization of data artifacts and steps. Steps 1440 and 1442 enhance the user interaction with digital models and digital workflows by replacing copy & paste with grab & drop mechanisms and reordering mechanisms. Step 1444 provides additional context such as dependencies on parent digital models and related versions, and further enhances the user's ability to organize specific data artifacts to check if the selected data artifacts are suitable for live documents. In some implementations, with a multimodal interface, the user can select and slide data artifacts to display the related parent model. The user can also expose the order of digital models providing the artifacts by selecting the entire sequence of artifacts in the live document and gesturing with slides, double taps, and double clicks. Such gestural interactions help the user review the order of artifacts in the context of the sequence of related digital models and intuitively make decisions.

[0262] Visual comment Some implementations of the multimodal interface by IDMP offer visual commenting in addition to text-based commenting. Visual commenting in a virtual environment allows users to interact with and annotate data artifacts by combining voice, motion, and visual cues, significantly improving collaboration and contextual clarity. This feature allows users to send comments not only through text but also through gestures and actions in the virtual space using avatars. For example, users can interact with CAD designs and record comments, such as selecting specific parts, pointing to details, or explaining their thoughts with voice and hand movements. This approach extends traditional commenting methods by incorporating dynamic interaction, enabling collaborators to more intuitively indicate problems and suggestions. Visual commenting also includes zooming in on parts, comparing elements, and displaying contextual relationships, all within a virtual environment. This method provides a richer, more interactive way of delivering feedback, making it easier for collaborators to understand and address specific points, ultimately improving the efficiency and effectiveness of the collaborative process. In exemplary implementations of simulations using spatial computing interfaces, recording visual and spatial audio comments simultaneously with specific runs or time points within the simulation can further enhance the context for other collaborators to understand the commentary.

[0263] Another example of visual commentary is providing visual commentary while running fluid simulations on a digital thread. For example, in a virtual environment, a simulation can be run to visualize airflow in 3D space as a user examines an airfoil. While the simulation is running, the user can highlight changes in airflow and explain them in real time. The simulation and commentary are recorded synchronously. The user's commentary can also be recorded with spatial audio, allowing the virtual avatar itself to be positioned within the 3D space of the airflow, and the spatial audio can further capture the direction the user is pointing within the simulation. This synchronous recording improves the ability to convey complex information, allowing collaborators to gain a deeper understanding of the subtle nuances of the simulation and the meaning of the observed changes. This type of visual commentary makes simulation reviews and discussions more comprehensive and interactive, leading to more informed decision-making and improved collaborative outcomes.

[0264] Examples of voice-to-gear Fig. 15 In accordance with one embodiment of the present invention, we demonstrate the exemplary generation and execution of an orchestration script through a voice / conversation interface. The first step shows the data processing of the user interface. In step 1502, the user sends a voice command containing specific modeling and simulation parameters. In step 1504, the system saves the voice file containing the user's input to a database. In step 1506, the system converts the user's input into a voice-to-text command and identifies the parameters therein using a machine learning model. For example, a transformer-based language model such as LaMDA ("Language Model for Conversational Applications") may be used in this step. LaMDA was introduced by Collins et al. "LaMDA: Our Breakthrough Conversational Technology" (2021), available blog.google / technology / ai / lamda / , the full text is incorporated here by reference.

[0265] Step 1508 generates a contextual AI based on LLM (1512) that generates inference for a digital thread (1508) based on user input, and then generates a prompt for the syntactic AI (1510).

[0266] One concrete example is prompt engineering (1510). First, the context AI is executed, and then the context AI uses the LLM-defined system (1512) to infer the digital thread (15). After some fine-tuning, the context AI presents user input in a standard format, each presenting a set of DE tasks associated with a model or tool. For example, the inferred digital thread might be "validate the requirements of a SysML model by comparing them with static and dynamic analysis using an FEA tool in a CAD model of an airplane wing and a low-resolution simulation mesh."

[0267] In step 1514, the Syntax AI develops a relevant parameterized script for all digital models within the identified digital thread. In some implementations, the parameters are hidden from the Syntax AI and then added later by a software module (e.g., a Python script) or another LLM.

[0268] The last three steps are performed on the DE platform application side (see diagram). 1) In step 1516, the generated script with the design parameters is executed and a new CAD design is generated. 2) In step 1518, the generated CAD design is saved. 3) Finally, in step 1520, the saved CAD design can be viewed.

[0269] Examples of spatial computation Fig. 16 In accordance with one example of the present invention, we demonstrate the exemplary generation and execution of an orchestration script through a spatial computation interface.

[0270] The first step demonstrates data processing for the user interface. In step 1602, the user makes gestures, points to real objects, or selects virtual objects in the user interface (UI). In step 1604, the system interprets the user's input into a specific command. The user can provide feedback to refine the system's interpretation process. In step 1606, the system uses a machine learning (ML) model to convert the user's input into a text command and identify its parameters.

[0271] Step 1608 generates a contextual AI based on an LLM or Transformer (1612), which generates inference for a digital thread (1608) based on user input, and then generates a prompt for a syntactic AI (1610).

[0272] One concrete example is prompt engineering (1610). First, the context AI is executed, and then the context AI uses the LLM-defined system (1612) to infer the digital thread (16). After some fine-tuning, the context AI presents user input in a standard format, each presenting a set of DE tasks associated with a model or tool. For example, the inferred digital thread might be "validate the requirements of a SysML model by comparing them with static and dynamic analysis using an FEA tool in a CAD model of an airplane wing and a low-resolution simulation mesh."

[0273] Another implementation example is contextual AI (1612), which converts user input into a format that syntactic AI can process, such as tokenized representations or vectors that encode and interpret the input. In such implementations, the representation is mediated by an intermediate transformer prompt generation (16), whose purpose is to encode user input to make it easier to interpret and process in subsequent stages of the AI ​​toolchain.

[0274] In step 1614, the Syntax AI develops a relevant parameterized script for all digital models within the identified digital thread. In some implementations, the parameters are hidden from the Syntax AI and then added later by a software module (e.g., a Python script) or another LLM.

[0275] The last three steps are performed on the DE platform application side (see diagram). 1) In step 1616, the generated script with the design parameters is executed and a new CAD design is generated. 2) In step 1618, the generated CAD design is saved. 3) Finally, in step 1620, the saved CAD design can be viewed.

[0276] Code interface for bot-user interaction As shown below and explained in conjunction with the diagram, the bot communication interface (1700) integrates multiple modules to streamline and secure data exchange. These modules handle TCP / IP and UDP communication (1702), socket-based communication (1704), and API requests (1706) or embedded-based (1706) in text formats (JSON, Code, XML, etc.). This system also facilitates VoIP communication (1710).

[0277] The main implementation steps include the deployment of a Learning Management System (LMS) for text processing. For VoIP communication, it processes the audio stream and converts it to text. It also maintains continuous message listening between TCP and UDP protocols.

[0278] Incoming web requests are processed efficiently, and the system includes implementations of webhooks that trigger specific responses to these requests.

[0279] Finally, security measures are of utmost importance. The system implements attribute-based access control (ABAC) to regulate access to resources. Additionally, it maintains auditability, manages reliable records of operations and activities, and conducts tracking and reviews. The combination of these functions results in a robust, secure, and efficient communication interface for bots.

[0280] The implementation steps of the bot interface 1700 include the following: 1. Definition of communication channels: Develop interfaces corresponding to each type of communication channel. This includes APIs (RESTful), VoIP communication, text messages, sockets, and other potential media. 2. Data processing: 1. Implement a language model (LLM) that can process various data such as JSON, code, XML, etc. Implement natural language understanding (NLU) and natural language processing (NLP) capabilities to handle multiple dialects. 2. For text data, add an additional processing layer with an LLM using specific approaches such as punctuation processing and sentiment analysis. 3. In the case of Voice over IP, implement functions to process the audio stream, convert the audio to text, and process it using timestamps. 3. Protocol implementation: a. Create different communication protocols to handle different types of messages. Implement TCP if the order is important and UDP if the order is not important. b. Include HTTPS for secure communication 4. How to handle connections: 1. Implement a socket connection for real-time interactions. This interface should keep the connection open and continuously receive messages. 2. For APIs, create standard request-response interactions. 5. Request handling: Implement GET request processing and webhook management functionality. For more direct interaction with the model, provide an interface that supports embedded interactions. 6. System adaptability and security: a. To increase the flexibility of bot communication, let's create a system interface that can adapt to different dialects. b. Error detection and recovery c. Implement robust security measures, including encryption of data in transit and while stationary, secure user authentication, and regular security audits. 7. Test: Implement comprehensive unit tests, integration tests, load tests, and stress tests to ensure that all parts of the system are functioning as expected and to identify potential security vulnerabilities.

[0281] The main components of the code interface include: 1. Communication Channel a. API b. Voice over IP c. Text messages d. Socket e. Other general media 2. Handling different data types a. JSON, Code, XML: These are processed using a Language Model (LLM) that can handle multiple languages. b. Text: You can add additional LLM boxes for text processing. c. Audio: VoIP processes the stream, converts the audio to text, and processes it using timestamps. 3. Protocol Options a. TCP: Guarantees message ordering appropriate for specific use cases. b. UDP: Does not guarantee message order and can be used for other specific purposes. 4. Connection Type a. Sockets: Ideal for real-time connections, providing open and continuous connectivity. b. API: Typical of non-real-time standard request-response interactions. 5. Request Processing a. Obtain requests: Request to process requests and manage webhooks. b. Direct interaction: Possible in the model using embeddings 6. Adaptability and Security The system's ability to adapt to different dialects provides flexibility in communication between bots, and each interaction is authenticated, tracked, and auditable.

[0282] As previously stated, Fig. 17, in accordance with the examples of the present invention, an example flowchart illustrating aspects of the operation of the disclosed system relating to multimodal communication of the code interface is shown. Part of the figure in the diagram represents various ways in which the bot communication interface integrates multiple modules to streamline and secure data exchange. In particular, these modules include a TCP / IP and UDP communication module (1702) for handling TCP / IP and UDP communication, a socket-based communication module (1704) for handling socket-based communication, a text-based API module (1706) for handling text-based formats (JSON, Code, XML, etc.) and an embedded-based request module (1710) for handling embedded-based requests via an embedded-based request module. The system also includes a VoIP module (1710) for facilitating VoIP communication.

[0283] The various modules described above may be managed through a digital engineering platform (1712). A digital platform such as IDEP or more commonly IDMP may be implemented to enable various enhancements to machine learning and additional features such as the security and data privacy measures disclosed herein. For example, the various modules described above may include implementation capabilities for incorporating a learning management system (LMS) for text processing. In some aspects, in VoIP communication, the system may process one or more audio streams and convert them to text. It also maintains continuous listening of messages across various protocols, such as TCP and UDP.

[0284] In other aspects, incoming web requests are processed by, for example, a system implementation of webhooks, which triggers specific responses to these requests.

[0285] Finally, the system incorporates specific security measures as part of its overall structure. For example, the system implements attribute-based access control (ABAC) to restrict access to resources. Furthermore, the disclosed system maintains auditability by keeping a reliable record of operations and activities, which can then be tracked and reviewed. These features, when combined, create a robust, secure, and efficient communication interface, for example, for bots to communicate with each other.

[0286] Exemplary GUI / API Interface: AI-Assisted Requirements Verification Fig. 18 Based on several examples of the present invention, a descriptive flow diagram is shown for exemplary use cases in which a GUI / API interface is used in an AI-assisted requirements validation process.

[0287] In this example, users can upload digital model files (e.g., CAD files for airplane seats) to IDEP via a GUI or API interface. The CAD files are in .zip format and contain the entire seat assembly, and users can view a 3D view of the design through the GUI to verify that the correct files have been uploaded. The same GUI may also be used for further user input or instructions to validate seat requirements.

[0288] Next, users can upload requirements files.1806 For example, a user can click the "Upload Requirements" icon to start the upload process and select the Excel requirements document to upload. The DE system may convert the Excel file to CSV format. Requirements describe the necessary functions and characteristics of a system during design, implementation, and operation. Thus, requirements set constraints and goals in both the design space and the objective space, and trade off design characteristics and limitations such as performance, schedule, cost, and lifecycle characteristics.

[0289] After processing, a list of requirements extracted from the requirements file is displayed to the user, allowing them to walk through it. Users can modify individual requirements as needed. In some implementations, the DE platform may display error messages to the user if it automatically detects potential errors or conflicts.

[0290] Next, users can begin the AI-assisted requirements validation process using the GUI. The validation process workflow is displayed to the user, allowing them to monitor the validation progress, verify items correctly, review example error lists, and provide feedback to the system if needed.

[0291] Report 18 may be automatically generated by the DE platform once verification is complete. The DE platform may also provide features for tracking and archiving verification history, as well as the ability to share reports via downloadable links.

[0292] Exemplary voice control interface Fig. Following the examples in Disclosure 19, an example workflow is shown illustrating aspects of the operation of the disclosure system related to a voice-controlled interface. In a sense, the flow represents aspects of user interaction with a CAD model using various interfaces (e.g., VR / AR / voice / text interfaces, etc.). In step 1902, the user can upload a model (e.g., a sheet CAD model) to the DE app. In some aspects, it may also be possible to upload specific file formats, such as zip files containing assemblies (code). In other aspects, the system may provide a 3D view of the model in a graphical user interface (GUI) and confirm with the user that the correct file has been selected before uploading.

[0293] In step 1904, users can view CAD models using VR / AR. Specifically, users can connect and activate VR glasses on the platform. The user can then select a model to view in VR. The system can display the selected model and related information extracted from the file.

[0294] In Step 1906, users can use voice input and chat input through the user interface. Specifically, the system may display a menu of command options available via voice or text. The user can then select and send the desired command via voice or text.

[0295] In step 1908, the voice command processing tool is launched and begins functioning. Specifically, commands are converted to text, displayed to the user, and confirmed. Furthermore, the system may use an AI model to generate and present user suggestions related to the commands and the model, prompting user consideration.

[0296] In step 1910, the system may sequentially describe use cases for one or more requirements for a particular design. In some sense, this can be achieved through VR, text, audio, etc. In particular, the system lists the total number of requirements, including qualitative requirements. Furthermore, the user can modify the requirements as needed.

[0297] In step 1912, the system can begin the generation process to produce the updated output. In particular, the system can use AI to help produce a better output than the unassisted case by comparing it with preconditions. Specifically, this generation process begins with the user making a selection (e.g., clicking the "Start" button).

[0298] In step 1914, the system may display output on a screen or other display. As mentioned earlier, the system may use AI to help generate better output than the unassisted case by comparing it with predetermined conditions. Furthermore, the system may provide the latest output via a downloadable link. Additional system features in this step may include, but are not limited to, history viewing and archiving functions.

[0299] In step 1916, the system can be configured to allow users to view and compare new outputs with previous versions. In particular, all the latest information, including model size and dimensions, can be viewed.

[0300] Fig. Following the examples of disclosure 20, another workflow example is shown that illustrates aspects of the operation of the disclosure system related to the voice-controlled interface. In a sense, the flow represents aspects of user interaction with CAD models using various interfaces (e.g., VR / AR / voice / text interfaces). Generally, as detailed below, the user connects VR glasses to the platform and selects a display model. The system displays the model in VR using both the requested data and the CAD file. Furthermore, when interacting with the 3D model, the system highlights any inconsistencies or errors and suggests corrections. Step 20 allows the user to upload a model (e.g., a sheet CAD model) in the DE app. In particular, the user can upload predefined types of files (e.g., a zip file containing an assembly). The system may display a 3D view in the GUI to verify that the correct file has been selected.

[0301] Step 2004 allows users to upload requirements. Specifically, users can click the "Upload Requirements" button, select a requirements document (e.g., Excel document), and convert it to a specific format (e.g., CSV format).

[0302] Step 2006 can receive voice and chat input. Specifically, users can input voice or chat commands through the UI. The system can then present voice or text responses to the user. Furthermore, users can send selected voice or text commands through the interface.

[0303] In Step 2007, users can view CAD models using VR / AR. Specifically, users connect VR glasses to the DE platform and select a VR display model. The system then presents the model in VR, including related information. Users can then interact with the system using voice commands to request changes between files or perform cross-checks.

[0304] Step 20 of the system includes a voice command processing tool, which is used for activation and functionality. Specifically, voice commands are converted to text, displayed to the user for confirmation, and the AI ​​model generates and presents user suggestions.

[0305] In Step 2010, the system may guide the user through one or more requirement use cases. At some point, the system lists the total number of requirements, some of which are qualitative. Furthermore, requirements may be modifiable as needed.

[0306] In step 2012, the system can begin the verification process. Specifically, the system may allow the user to click the "Verify" button, select the CAD model, select all the corresponding requirements, and then click the "Selected" button to begin verification.

[0307] Step 2014 may allow for a verification process that involves feedback from human experts. Specifically, users can monitor the progress of the workflow, verify items that have been correctly validated, and review examples of error lists.

[0308] In Step 2016, the system may provide digitally signed reports. Specifically, the system may provide a report describing what was done, and that report may be provided in the form of a downloadable link. The system may also include additional features such as history and archiving capabilities.

[0309] Step 2018 introduced a system that allows users to view updated outputs of CAD models through VR / AR and compare those updated outputs with requirements.

[0310] At one or more of the steps described above, the user connects VR glasses to the platform and selects a display model. The system displays the model in VR using both the requested data and CAD files. As the user interacts with the 3D model, they receive system highlights of any discrepancies or errors and suggestions for corrections.

[0311] AI-assisted conversational interface Examples of requirements verification Fig. 21 Based on several examples of the present invention, a descriptive flow diagram of an exemplary AI-assisted requirements validation process using an interactive interface is shown. In particular, input requirements files may be parsed using a transformer or a language learning model (LLM). Preprocessing before running the AI-assisted requirements validation process 2102 can be completed to add embeddings from reference requirements documents (e.g., MIL-HDBK-516C aviation compliance certification standard, manned and unmanned, fixed-wing and rotary-wing aircraft systems) to the LLM.

[0312] At the start of the AI-assisted requirements validation process, requirements files (e.g., in Excel or CSV format) and corresponding digital model files (e.g., CAD) may be uploaded and compared against the requirements.2104

[0313] The requirements file is splicable.2106, converted to Model Splice R, and individual quantitative or qualitative requirements are extracted using a dedicated requirements model splicer. Model Splice R is further processed using pre-processed LLM for evaluation, classification, or classification of the qualitative and quantitative requirements.

[0314] Next, the user gives instructions in a conversation (voice / text) 2108 targeting specific requirements. In response, the LLM suggests actions based on the user's input 2110. The user can choose the suggested action or provide further instructions 2112. Because the LLM is trained with embeddings from previous reference requirement documents and feedback from other users, a feedback loop with the user leads to the refinement of the LLM's suggested actions.

[0315] For all selected actions, a model splice R is processed based on user instructions regarding specific requirements. For each selected requirement, the input CAD model is spliced ​​into a model splice M, which can implement the user-selected action for that specific requirement. This initiates an AI-assisted requirements validation process. If a model splice M already exists, it can be updated based on the action selected by the user for that specific requirement.

[0316] Next, model splice R and model splice M can be appropriately linked, and each corresponding requirement of splice R can be evaluated against the model parameters of splice M to verify the satisfiesability of the requirements and outputs.2118 A human expert can review, verify, and approve the results of each requirement verification,2120 and a verification report21 can be generated after all requirements have been considered.22

[0317] Example of digital thread correction Fig. Following the examples in Disclosure 22, an example workflow is shown illustrating aspects of the operation of the disclosure system related to the AI-assisted conversation interface. In step 2202, the system is configured to allow the user to upload a digital model via GUI or API call 2230. In step 2204, the system is configured to create the appropriate model splice. In step 2206, the system is configured to receive user input (voice / text) on the analysis and control plane.

[0318] In step 2208, the system processes user input as speech and text through an algorithm. As part of this step, the system may receive speech / text / sentiment analysis.

[0319] In step 2210, the system is configured to provide a menu of platform-specific actions that the user can take. As part of this step, the system may receive and provide descriptions of digital threads and digital engineering (DE) tool-specific actions that the user can view and perform. Furthermore, the user can interact with the system through interactive voice response (IVR) or a chatbot and feed the results of that interaction back into the user input of step 2206.

[0320] In step 2212, the system is configured to create an orchestration script based on user input. As part of this step, the system can receive software tool-specific scripts and parameter substitutions. Software tool-specific scripts can be received through the Syntax AI module 2224. Parameter substitutions can be received through the Context AI module 2222.

[0321] In step 2214, the system is configured to execute one or more corresponding orchestration scripts. These scripts can automate the execution of related tasks and subtasks to achieve specific outputs requested by requests or commands. In step 2216, the system is configured to report user input and performed actions with digital signatures. Specifically, the system may provide a report describing what was done, and that report may be provided in the form of a downloadable link. The system may include additional features such as history and archiving capabilities.

[0322] Exemplary graphical user interface Exemplary GUI for digital artifacts within a digital thread Fig. 23 According to one embodiment of the present invention, a screenshot of an exemplary graphical user interface (GUI) for manipulating digital threads on IDEP is shown. The GUI provides users of the Interconnected Digital Engineering Platform (IDEP) with the ability to select and view digital artifacts, including initial, latest, and intermediate versions, to which they have access rights. Fig. 23 Displays the header of the browser window 2302 which contains digital thread links for easy navigation. Below the header are domain and security level banners 2304 which indicate the domain, platform software version, and security level, allowing the user to understand the domain and security protocol on which they are operating. Security level indicator 2306 displays the user's maximum security access level within the platform (e.g., "Level 1"). The security level indicator is referred here synonymously with "info security tag", "infosec tag", or "info sec tag".

[0323] The interface also includes a search bar,2312 which allows users to comprehensively search digital engineering models, files, and documents across platforms through IDEP, facilitating efficient information retrieval across the platform. Adjacent to this are user and domain fields,2310 which provide the user's domain information (e.g., client name). The user and domain fields allow access to login, user profiles, and subscription information.

[0324] The GUI's top menu has additional features. For example, the Digital Artifact Name field 2320 shows the name of the digital model or document, and may include its version. Furthermore, the Digital Thread Artifact field 2326 displays the digital artifact name. The Digital Artifact Security Level Indicator 2322 shows the security level (e.g., "Level 1") of the digital artifact being accessed. In one example, an expandable security level menu 2322 is used next to the Digital Artifact Security Level Indicator, allowing the user to select the target security access level of the digital artifact, thereby filtering to show only the portion of the digital artifact accessible at a specific security level. In another example, the user may use the Digital Artifact Security Level Indicator 2322 to down-select the security level while sharing the digital artifact, so that only the portion of the digital artifact corresponding to the specified security level (e.g., only security access levels below the user's security level (e.g., "Level 1" in the diagram)) will be viewable and shareable by the user. Button 2324 on the user interface includes options such as copying the digital artifact link, opening the comments section, accessing digital artifact information, managing shared access, and exporting the digital artifact.

[0325] In some implementations, granular dynamic information security tags (e.g., 2306 and 2322) are critical elements of digital threads and live digital object generation / display systems and their associated GUIs. Model splicers and IDEP systems enable granular dynamic information security tags 2306 and 2322. In some implementations, the IDEP's digital thread system updates against authorizations, licenses, and regulations using metadata from DE models and documents. In some implementations, granular dynamic information security tags 2306 and 2322 are dynamic and are updated before digital thread updates to ensure that the correct authenticated user has legitimate access to the digital artifacts and data for the update to be performed.

[0326] At the heart of the fig. 23. The Digital Artifact Viewer 2340 displays digital artifacts to which the user has access permissions at the appropriate information security level. Finally, regarding the right side of the fig. 23. The Version Panel 2350 displays the version history of digital artifacts within the digital thread. In the exemplary GUI of the figure 23. The Version Card 2352 indicates that the user is viewing the "latest" version of the digital artifact displayed in the viewer. The Version Card 2354 shows the option to select the "earlier" version of the digital artifact. In some implementations, all versions that the user can view at their information security level are accessible from the Version Menu in the Version Panel 2350.

[0327] Correcting digital artifacts is very likely to occur during the execution of digital threads related to complex DE tasks. The diagram shows an example of how the version control GUI23IDEP can track versions with appropriate security and access controls.

[0328] Exemplary GUI for orchestration scripts in digital threads Fig. 24 According to one embodiment of the present invention, a screenshot of another exemplary graphical user interface (GUI) used for manipulating digital threads on IDEP is shown. The GUI provides users of the Interconnected Digital Engineering Platform (IDEP) with the digital thread creation functionality described herein. Fig. 24 Displays the header of the browser window 2402 which contains digital thread links for easy navigation. Below the header are domain and security level banners 2404 which indicate the domain, platform software version, and security level, allowing the user to understand the domain and security protocol they are operating on. Security level indicator 2406 displays the user's maximum security access level within the platform (e.g., "Level 1").

[0329] The interface also includes a search bar, allowing users to comprehensively search across platforms for digital engineering models, files, digital threads, and documents through IDEP, facilitating efficient information retrieval across the platform. Adjacent to this are user and domain fields, providing the user's domain information (e.g., client name). The user and domain fields allow access to login, user profiles, and subscription information.

[0330] The GUI's top menu offers additional functionality. For example, the Digital Thread Name field 2420 displays the name of the digital thread and can include its version. The Digital Thread Security Level Indicator 2422 shows the security level of the digital thread being accessed (e.g., "Level 1"). In one example, an expandable security level menu 2422 is used next to the Digital Thread Security Level Indicator, allowing the user to select the target security access level for the digital thread, thereby filtering to show only the portion of the digital thread accessible through a specific security level. In another example, the user might use the Digital Thread Security Level Indicator 2422 to down-select the security level while sharing the digital thread or the live document corresponding to that thread, so that only the portion of the digital thread corresponding to the specified security level (e.g., only security access levels below the user's security level, e.g., "Level 1" in the diagram) will be viewable and shareable by the user. Buttons 2424 in the user interface include options such as copying the digital thread link, opening the comment section, accessing digital thread information, managing shared access, and exporting the digital thread.

[0331] In some implementations, granular dynamic information security tags (e.g., 2406 and 2422) are critical elements of digital threads, live document systems, and their associated GUIs. Model splicers and IDEP systems enable granular dynamic information security tags 2406 and 2422. IDEP's digital thread system uses metadata from DE models and documents to match and update against authorizations, licenses, and regulations. In some implementations, granular dynamic information security tags 2406 and 2422 are dynamic and are updated before digital thread updates to ensure that the correct authenticated user has legitimate access to the digital artifacts and data for the update to be performed.

[0332] As mentioned earlier, a digital thread is a set of orchestration scripts for selectively exchanging data between documents and DE model files. Therefore, a digital thread links all resources involved in achieving a particular DE task, including each section of the orchestration script, the associated DE model, and any relevant contextual information and metadata.

[0333] For secure organization and navigation of digital threads, please refer to the GUI in the diagram. The Digital Thread Outline Viewer features a 24-bit digital thread outline viewer function located on the left side of the 24-bit 30-bit fig. It provides links to individual sections of the digital thread, including code blocks that perform individual subtasks within the orchestration script, and text blocks that provide contextual, parametric, requirements-related, and authentication-related information about the linked DE model. Text blocks may also include text paragraphs, orchestration code, comments, and data sources. Within the Digital Thread Outline Viewer, the Digital Thread Details Viewer shows the sections of the secure digital thread, the associated digital engineering (DE) model, related live documents, source IT domains, and last update timestamps, each tagged with the appropriate information security level (e.g., "L1" or "Level 1"). In some examples, the information security tags on code blocks indicate execution restrictions for those code blocks; that is, the code block can only be executed by user entities with an equivalent or higher information security level. In some implementations, information security tags may indicate read privileges, and code blocks can only be presented and viewed by user entities with an equivalent or higher information security level.

[0334] In some cases, if a section of a secure digital thread contains content requiring a higher level of security for viewing, the user may be presented with the option to request access. If the user requests such access, an authorized user with higher security privileges will be notified and the request will be verified. In other cases, if a section of a digital thread contains content requiring a higher level of security for viewing, the section will not be displayed, and no prompt for access request will be provided.

[0335] At the heart of the fig. 24. Section Viewer 2440 displays the contents of each secure digital thread section, ensuring that all orchestration script code, code comments, and text blocks are updated based on the data of the linked DE model. Model data and associated security access may be provided by the aforementioned model splicing. Finally, on the right side of the fig. 24. Comment Panel 2450 displays digital thread comments and may include comment sharing and resolution functions.

[0336] Exemplary LiveBoard management Fig. 25 According to one embodiment of the present invention, an exemplary graphical user interface (GUI) for generating or updating live suites and collaboration boards on IDMP is shown. Fig. 25 in particular displays the header of the browser window 2502 which contains live board links for easy navigation. Below the header are banners for the domain and security level 2504 which indicate the domain, platform software version and security level, allowing the user to understand the domain and security protocol they are operating on. A security level indicator 2506 displays the user's maximum security access level within the platform (e.g., "Level 1").

[0337] The interface also includes a search bar, allowing users to comprehensively search across platforms for digital engineering models, files, digital threads, and documents through IDEP, facilitating efficient information retrieval across the platform. Adjacent to this are user and domain fields, providing the user's domain information (e.g., client name). The user and domain fields allow access to login, user profiles, and subscription information.

[0338] The GUI's top menu offers additional features. For example, the Live Board Name field 2520 displays the name of the live board and can include its version. The Live Board Security Level Indicator 2522 shows the security level of the live board being accessed (e.g., "Level 1"). In one example, an expandable security level menu is used adjacent to the Live Board Security Level Indicator 2522, allowing the user to select a "view" of the target security access level of the live board, thereby filtering to show only the portion of the live board accessible at the specified security level. In another example, the user may also use the Live Board Security Level Indicator 2522 to downsect the security level while sharing the live board or its associated live document, sharing only the portion of the live board corresponding to the specified security level. Only security access levels below the user's security level (e.g., "Level 1" in the diagram) will be viewable and shareable by the user. Buttons 2524 in the user interface include options such as copying the live board link, opening the comment section, accessing live board information, managing shared access, and exporting digital threads.

[0339] In some implementations, granular dynamic information security tags (e.g., 2506 and 2522) are critical elements of live board and live document systems and their associated GUIs. Model splicers and IDEP systems enable granular dynamic information security tags 2506 and 2522. IDEP's digital threading system uses metadata from DE models and documents to match and update authorizations, licenses, and regulations. In some implementations, granular dynamic information security tags 2506 and 2522 are dynamic and are updated before live board updates to ensure that the correct authenticated user has legitimate access to perform or view updates to the digital artifacts and data.

[0340] Fig. On a 25 Live / Magic Suite collaboration board, individual tile-like artifacts are represented by Live or Magic Chips. Magic Chips can also be invoked from other third-party software tools such as Google Suite and Slack.

[0341] As mentioned earlier, the live board represents data managed through a digital thread that selectively coordinates data exchange between document and DE model files. Thus, the digital thread links all resources involved in achieving a particular DE task, including each section of the orchestration script, the associated DE model, and relevant contextual information and metadata.

[0342] Live / Magic Links and Live / Magic Chips In exemplary live suites, collaboration boards, or live documents, a magic link (or live link) refers to a hyperlink that points to an artifact. In Magic Docs, links are used to dynamically connect artifacts within a document. In collaboration boards, one or more magic links can be used as live links to digital artifacts in customer data storage. These links allow updates and contextual information to be displayed directly within the document, ensuring users always have access to the latest information.

[0343] When LiveLink is executed, it is presented in a tile format called Live Chips or Magic Chips, as shown in the figure. 25. Figs. 25 Magic Chips for vehicle performance artifacts 2540, body block diagram 2542, and isometric projection diagram 2544.

[0344] Magic Links are also represented within blocks in Magic Docs and within collaboration boards in Magic Suite. Through the Magic Link in IDMP, the actual content of the artifact is not stored; rather, a link to the digital artifact itself is provided, ensuring that security, access, and content are updated each time the document or collaboration board is loaded. An example of how Magic Link data is represented within a Magic Document is shown in Table 1. JPEG2026528738000002.jpg194159

[0345] As a related example, Magic Chips are implemented as portable components, within third-party software such as Google Suite, Slack, and JIRA, to authenticate users and generate artifacts similar to Magic Links. Magic Chips can also operate outside of typical IDMP platforms, but only if user authentication (seamlessly linking with IDMP within the third-party software) is performed at both ends to allow access to the data.

[0346] As a related example, the Fyber platform uses tripod OAuth (OAuth 2.0) authentication to seamlessly connect with other online platforms such as Google Suite and Slack. This method allows users to grant the Fyber platform access to data on other platforms without sharing their credentials, and vice versa. Furthermore, users can grant other platforms access to their data and features through the Fyber platform.

[0347] This section provides a simplified overview of the process for integrating Fyber with other platforms, allowing Fyber to access user data on those platforms. Reversing the steps in this process would allow users to grant third-party platforms access to data on the Fyber platform. A. Registration: The Fyber platform registers with an OAuth provider (e.g., Google) and obtains a client ID and secret. B. Authorization Request: The Fyber platform directs the user to the OAuth provider's authorization page and grants access. C. Authentication Code Exchange: Once the user grants access, the OAuth provider redirects the user to the Fyber platform along with an authentication code. The Fyber platform then exchanges this code for an access token. D. API Requests: The Fyber platform uses access tokens to request and execute data manipulations from OAuth providers on behalf of the user (in accordance with the permissions of the access token).

[0348] In some implementations, this process is implemented within a plugin (e.g., a browser plugin) that can handle the OAuth flow, token management, and API requests. This integration enables seamless and secure data access to trusted sources using the Fyber platform across multiple platforms.

[0349] Secure breakout rooms for ZT collaboration Some implementations of the Integrated Digital Model Platform (IDMP) facilitate zero-trust (ZT) collaboration through breakout rooms and secure compartments within the same live space, document, and board. This is achieved by dynamically controlling access to data artifacts and related communications based on user and security network permissions. This secure compartment or breakout feature allows multiple users to work simultaneously on shared live documents and live spaces, with sensitive information limited to authorized individuals. For example, when digital models or artifacts are the focus, users without the necessary permissions are automatically excluded from viewing and listening to the relevant content. This exclusion can be achieved by muting audio channels for unauthorized users or visually masking artifacts. Similar to traditional breakout rooms, this concept ensures seamless and secure collaboration within a virtual environment using IDMP. Users can interact with shared spaces with peace of mind knowing that access to sensitive data is tightly controlled. This feature maintains security and privacy, allowing users to focus on their work without worrying about unauthorized data leaks.

[0350] Exemplary embodiment of multimodal operations Fig. 26 illustrates the use of a multimodal interface for accessing data through a virtual live board, following the example shown here. Fig. 26 specifically depicts a scene set in a virtual space "live space" where a human user is present, wearing a virtual reality (VR) headset, an augmented reality (AR) headset, or a spatial computing headset. Fig. 26 plays a role in collaborating with aircraft and engine design and simulation through an Integrated Digital Model Platform (IDMP). User 2602 invited two collaborators, depicted as virtual avatars (2606 and 26), into this immersive environment. Surrounding the humans are two virtual artifact walls (e.g., 2610). The virtual wall behind the user houses an artifact navigation panel (2614) that can be selected and displayed collaboratively with the two virtual avatars (2606 and 26). Additionally, the virtual wall behind the user houses an artifact sort panel (2615) that can be brought into and dropped into the live space or any virtual space available to the user (2602). The virtual environment may utilize multiple virtual spaces that the user can minimize, maximize, switch between, and swap. These may include virtual 3D spaces, virtual 2D walls, virtual 2D screens, virtual windows, and virtual desks (e.g., 2612).

[0351] Above it is a human user 2602 who uses hand gestures 26 captured with a spatial computing headset 302604, virtual reality (VR) gloves 2632, and a camera (not shown in the figure). 26) or with the spatial computing headset, they interact with a 3D model of an airplane 26 linked through IDMP. 16 This 3D model 2616 is displayed in a virtual space behind the dashboard, called the Live Board 2618, which contains various digital artifacts and separate magical documents 2620 with a digital thread of specific reviews by a team including the user and collaborators (2606 and 2608). In one specific example, the 3D model of the airplane 2616 is clearly presented to the user by visual distinctions such as a thick outline, as shown in the figure. 26. In one embodiment, there are other collaborators (not shown in the figure). 26) may collaborate on teamwork in a conversational format.

[0352] In the figure, Team 26 discusses aircraft design models and simulations. Fig. 26 shows a visualization of the airflow around the engine from a CFD analysis 2622 and data artifacts such as line traces of the lift coefficients 2624, both of which are linked to the simulation model. An example of the simulation model 2626 can also be displayed on the live board. Both the line trace data artifact 2624 and the airflow visualization 2622 are derived from the same simulation model 2626, which provides a comprehensive understanding of the aircraft's performance.

[0353] As a human user, 2602 reviews the aircraft's performance and virtual avatar (actively reviewing and commenting on additional elements of the 2606 and 26 engine design, and comparing them against a review checklist). The virtual avatar on the right, 2608, may not have permission to view the engine design of the virtual avatar on the left, 2606, which 2606 has seen. This access restriction may be communicated to the human user. Through the graying of the virtual avatar, 2608 indicates that inaccessible digital artifacts, models, threads, or documents are selected by the human user (as shown in the figure). 26. Access to elements of the browsing data is restricted for collaborators, 2608; digital artifacts, models, threads, and documents outside the security level are grayed out or obscured from virtual walls, spaces, windows, and live digital objects. User 2602 may also not have permission to view some of the digital artifacts or digital models owned by the user, such as artifacts in the operational and navigation panel, 2614, or the sorting panel, 2615. In some implementations, the platform may indicate this to the user. User artifacts, models, subsystems, and systems 2602 are not permitted to be accessed or manipulated through grayout, dashing, shading, censorship, coloring, or other obfuscation means.

[0354] Fig. 26 demonstrates how such a collaborative configuration using a multimodal interface enables a detailed and interactive review process in a zero-trust environment, facilitating effective communication and decision-making among team members, even if some members do not have access to all deliverables and design information.

[0355] Fig. 27 A flowchart illustrating the process of interacting with a live digital object is shown here, following the example provided. The process begins in step 2720, when the system receives a live digital object. A live digital object may include digital artifacts extracted from a digital model file through a model representation. A model representation may include locators to model type-specific digital model data and metadata. Next, in step 2730, the system initiates a connection to a multimodal interface. The multimodal interface can be configured to receive input from at least two different modalities and provide output. At least two different modalities may include at least a conversational modality and a spatial modality. In step 2740, the system receives the security level of the first user. In step 2750, based on the security level of the first user, the system determines the initial access rights to the digital artifact and the modification rights of the first user to modify the digital artifact. In step 2760, the system outputs the digital artifact to the multimodal interface through the connection, based on the permissions of the user who first accessed the digital artifact. In step 2770, the system receives one or more inputs through connections from a multimodal interface. These inputs include conversational inputs and spatial inputs from the first user related to the digital artifact. Finally, in step 2780, the system generates a modified digital artifact from the original digital artifact through a digital model representation based on the first user's permission to modify and one or more inputs. This completes the process. The flowchart in the figure describes the conversational and spatial interfaces, and similar techniques are applicable to any modality described herein and are within the scope of the present invention.

[0356] Fig. 28 A flowchart detailing the process of digital engineering using a multimodal interface is shown here, following the example presented. In step 2802, the system receives a first input of type 1 and a second input of type 2 through the multimodal interface. Type 1 and Type 2 may be different modalities. In step 2804, the system modifies the digital model representation that represents at least some of the digital objects in the digital model platform based on either the first or second input. In one embodiment, the digital object is a living digital object. In another embodiment, the digital object is a digital twin. In one embodiment, the digital object is a digital twin associated with a physical twin. Finally, in step 2806, the system provides feedback to the user based on the modification.

[0357] Examples of exemplary conversational interfaces One example of this disclosure relates to a system and method for providing a conversational interface to a digital engineering or digital modeling platform. This system and method allows users to interact with the digital engineering platform using text (natural language) or voice commands, improving user experience and efficiency. The disclosed system and method are merely examples of many other diverse implementations that constitute the application of the disclosure principles.

[0358] The first embodiment of an interactive interface to a digital engineering platform is a software application installable on a computing device. This software application includes a speech recognition module, a natural language processing module, and an interface module. The speech recognition module is configured to receive and interpret voice commands from the user. The natural language processing module is configured to understand the intent of the user's voice commands and translate them into commands that the digital engineering platform can understand. The interface module is configured to transmit these commands to the digital engineering platform and receive responses from the platform.

[0359] The initial implementation begins when a user issues a voice command to a software application. A speech recognition module receives and interprets the voice command. A natural language processing module understands the intent of the voice command and translates it into a command for the digital engineering platform. An interface module transmits this command to the digital engineering platform and receives a response from the platform. This response is returned to the user in a conversational format. This implementation offers the advantage of allowing users to interact with the digital engineering platform more naturally and intuitively compared to existing methods such as entering complex commands or using graphical user interfaces.

[0360] In some implementations, voice commands can be typed as plain text into a chatbot-like natural language interface.

[0361] The second embodiment of the interactive interface to the digital engineering platform is a hardware device including a microphone, speaker, processor, and memory. The microphone is configured to receive voice commands from the user. The speaker is configured to transmit responses from the digital engineering platform to the user. The processor is configured to run software applications stored in memory. These software applications include a speech recognition module, a natural language processing module, and an interface module, similar to the first implementation.

[0362] The operation of the second embodiment begins when the user issues a voice command to the hardware device. The microphone receives the voice command, the processor executes a software application to interpret the voice command, understand its intent, translate it into a command for the digital engineering platform, transmits the command to the digital engineering platform, and receives a response from the platform. The speaker communicates the response to the user in a conversational format. This implementation offers the advantage of being a standalone device that can interact with the digital engineering platform as an independent device without cloud connectivity.

[0363] The third embodiment of the interactive interface to the digital engineering platform is a cloud-based system. This system includes a speech recognition module, a natural language processing module, and an interface module, similar to the first implementation. The cloud-based system allows for remote processing and interpretation of voice commands, reducing the computational load on the user's equipment.

[0364] The operation of the third embodiment begins when the user issues a voice command to the cloud-based system. The speech recognition module receives and interprets the voice command. The natural language processing module understands the intent of the voice command and translates it into a command for the digital engineering platform. The interface module transmits this command to the digital engineering platform and receives a response from the platform. That response is returned to the user in conversational format. This implementation offers the advantage of reducing the computational load on the user's equipment and enabling more complex operations.

[0365] The disclosed system and method offer several advantages over existing conversational and voice interfaces. Firstly, it is specifically designed for digital engineering platforms, enabling the execution of complex operations and the use of technical terminology. Secondly, it allows users to interact with digital models and software tools more intuitively and efficiently, contributing to improved user experience and productivity. Thirdly, it is adaptive, learning from the user's voice and speaking style, and improving accuracy and efficiency over time.

[0366] Fig. 29 is an exemplary flowchart illustrating the process of digital engineering using a conversational interface, according to several examples of the present invention. In step 2902, the system interacts with the user using natural language based on voice and text commands interpreted through the conversational interface module. In step 2904, the system takes action through the digital model platform interface module based on the interaction with the user. Taking action on the digital model platform may involve modifying the digital model representation that represents at least a portion of the digital object. Finally, in step 2906, the system controls the conversational interface module and the digital model platform interface module via a processor. Note that the conversational interface can be combined with other multimodal modalities, and these combinations are also within the scope of the present invention.

[0367] alternative expression Next, we will describe various alternative manifestations. In one aspect or in one specific example, a system is provided for interacting with living digital objects, the system comprising program code that stores at least one processor and at least one memory. The program code is executable by at least one processor, and in order for at least one processor to execute its process, the program code contains code to perform the steps described above.

[0368] In one implementation, a digital review process is provided, and the program code includes code that sends the corrected digital artifact (corrected by the first user) to another user, receives instructions (approval or rejection) from the second user, and then further corrects the corrected digital artifact based on the instructions from the second user.

[0369] In one implementation, the program code also includes code to send the modified digital artifact to a second user, to receive a second security level from the second user and determine a second modification privilege for the second user, to receive a modification instruction from the second user, and to generate the modified digital artifact based on the modification instruction and the second modification privilege.

[0370] In one implementation, modifications and suggested modifications may be exchanged between a first user and a second user, each containing multiple digital artifacts. This implementation may include a second digital artifact accessed through a second model representation of the live digital object. The program code may also include code to receive a second user input to the live digital object from the second user, which may be conversational or spatial. The program code may also include code to further modify the second digital artifact based on the second user input, generating a second modified digital artifact.

[0371] Some implementations are configured to learn from specific users' voices and speaking patterns, improving accuracy and efficiency over time. In such implementations, the speech recognition module may be further configured to adapt to the user's voice and speaking patterns over time.

[0372] In some implementations, a live digital object is a live digital space that displays one or more documents or one or more applications on a 3D spatial display.

[0373] In some implementations, the digital thread controlling a ZT-compliant live digital object may include instructions to periodically (at frequent time intervals) verify that the user's access and modification permissions are current. The digital thread may also include instructions that condition updates to the digital artifact on verification of current access and modification permissions. Furthermore, the digital thread may include instructions to display the edited digital artifact if the user's permissions are no longer valid. In other examples, a typical live digital object may be updated without permission verification.

[0374] In some implementations, the system provides feedback to the initial user based on the correction of the digital artifact. In some implementations, the feedback may include a message indicating that the digital thread is broken or malfunctioning, or that the artifact cannot be modified for permission or other reasons. In some implementations, the feedback from the system includes errors and solutions to those errors related to the digital model.

[0375] In some implementations, the first user may receive comments from a second user regarding the digital artifact.

[0376] In one implementation, an NLP module is further configured to provide user feedback regarding the actions taken on the digital platform.

[0377] In one example, an ML model that has access to communication between the user and the digital platform is further configured to learn from interactions with the user and improve the execution of actions on the digital platform.

[0378] In one example, an interconnected digital platform includes a chatbot module that receives and interprets text prompts from users.

[0379] In one example, the program code also includes code that receives and interprets text prompts from the user.

[0380] In various implementations, a digital thread may contain a first and second orchestration script, with the first orchestration script being generated, modified, or executed by the first remote user, and the second orchestration script being generated, modified, or executed simultaneously by the second remote user.

[0381] Machine Learning (ML) and Neural Networks Machine learning (ML) algorithms are characterized by their ability to improve task performance over time without explicitly programming rules (i.e., learning) for performing the task. An ML model is the output produced when an ML algorithm is trained on data. Examples of the present invention include using one or more artificial intelligence (AI) and machine learning algorithms to interact with living digital objects and perform multimodal operations, including scripting, twinning, and model updates. The scope of the present invention includes a variety of exemplary ML algorithms. The following description illustrates modeling ML techniques for implementing various implementations of the present invention.

[0382] Neural Network A neural network is a computational model composed of interconnected units called "neurons" that work together to process information. It is a type of ML algorithm particularly effective for recognizing and predicting patterns based on complex data. Due to its ability to learn from large amounts of data and improve performance over time, neural networks are widely used in various applications such as image and speech recognition and natural language processing. Figure 30 describes the operating principle of a neural network based on an exemplary example of the present invention.

[0383] Fig. 30 shows a single-layer neural network, also known as a single-layer perceptron. The operation of a single-layer neural network involves the following steps:

[0384] 1. Input: Receive DE input vector TIFF2026528738000003.tif223004 and elements TIFF2026528738000004.tif33, j represents TIFF2026528738000005.tif315 ス The DE input is such that each element of the vector corresponds to an element in the 30th input layer. For an exemplary neural network model trained to update the IDEP script for multimodal operations, the DE input vector is TIFF2026528738000006.tif223004 may also take the form of a user prompt. DE input could be a user prompt, DE document, DE model, DE program code, system data from IDEP, or any form of data useful in digital engineering.

[0385] 2. Transfer function: The weights corresponding to each element of the DE input vector are multiplied together. TIFF2026528738000007.tif343008. These weighted inputs are summed up and combined as a transfer function to obtain the pure input to the activation function. TIFF2026528738000008.tif5203010. Each neuron in a neural network may have a bias value, which is added to the weighted sum of the inputs to that neuron. The weights and bias values ​​are learned during the training process. The purpose of bias is to provide all neurons with a trainable constant value, helping the model better fit the data. If bias is present, the net input to the activation function is TIFF2026528738000009.tif529. In the neural network model shown above (e.g., implementation of a script-updated ML model), the transfer function value of 3010 may represent the probability that an update of a certain script will be output.

[0386] 3. Activation Function: The activation function through which the pure input is passed determines the activation value, which is the output of the neuron. The activation function σ determines the activation value, which is the output of the neuron. It is usually a nonlinear function such as the sigmoid function or the ReLU (rectified linear unit) function. The threshold θ of the activation function is the value that determines whether the neuron is activated or not. For some activation functions, such as the step function, the threshold is a specific value. If the input exceeds the threshold, the neuron outputs a constant value, and if it is below the threshold, it outputs a zero value. For other activation functions, such as the sigmoid function and the ReLU (rectified linear unit) function, the threshold is not a specific value, but a transition point on the function's curve.

[0387] In the typical neural network model described above, the activation function σ3014 may be ReLU activated at a threshold θ3016, where θ3016 represents the minimum probability that a given script update will be implemented. Therefore, with activation function σ3014, an update of the given script is obtained when the implementation probability exceeds the threshold θ3016.

[0388] 4. Output: The activation value ο3018 is the output of the activation function. This value is passed to the next layer of the network, or in the case of the final layer, it becomes the final DE output. In the exemplary neural network model above (e.g., an implementation of a script-update ML model), there are multiple activation values ​​ο3018 that can be combined from multiple layers of the neural network to generate a text variable that represents the script update most likely to satisfy a particular DE input 3004. The DE output can also be an updated twin configuration, digital twin, physical twin, DE document, DE model, DE program code, or any form of data useful in digital engineering.

[0389] In the exemplary neural network discussion in the figure, examples of 30 specific script-update ML model implementations are shown using neural networks. Similar techniques are used in the implementation of model-update ML models, feedback ML models, and other NN-based components of the systems and subsystems described here.

[0390] Fig. 31 This shows an overview of the training process of an IDEP neural network based on an exemplary example of the present invention.

[0391] Training an IDEP neural network involves repeatedly updating weights and biases to minimize the difference between the predicted output and the true or target output. The predicted output is the result produced when a set of inputs from the dataset passes through the network. The predicted output corresponds to the DE output of the IDEP neural network. The true or target output is the true desired result. The difference between the predicted output and the true output is calculated using a loss function, which quantifies the error the network predicts.

[0392] The loss function is part of the cost function, and 3108 is an indicator of how well the network is performing across the entire dataset. The goal of training is to minimize the cost function, 3108. This is achieved by iteratively adjusting the weights and biases, with 10 being the steepest downward movement in the cost function. The magnitude of these adjustments is determined by the learning rate, a hyperparameter that controls the amount of change in weights and biases at each iteration. A lower learning rate results in smaller changes and slower convergence to the cost function's minimum, while a higher learning rate results in larger changes and faster convergence, but carries the risk of exceeding the minimum.

[0393] The IDEP neural network model 3102 is based on the exemplary neural network model discussed in the context of the above figure 30 (e.g., an implementation of a script-update type ML model), and was trained to determine whether to implement a specific script update for multimodal operations. ●Weights and biases 3110 are hyperparameters of the IDEP neural network, which are updated in each iteration of the training process, and are described in the context of the figure. 30, ●Prediction output 3104 is a binary prediction based on whether a particular script update will be implemented based on the sample multimodal requirements (or a normalized score ranking that prioritizes the order in which script updates are displayed to the user). ● True / Target Output 3106 is the correct decision (i.e., the sample's ground truth output) of whether to implement the given script update based on the sample's multimodal requirements. ● The loss function 3108 is the difference between the evaluation and the true output (e.g., a binary error indicating whether the IDEP neural network's judgment was correct). ● The cost function 3108 is the average error across the entire training dataset, including the sample multimodal requirements and corresponding implementations of the given script updates. ● The learning rate of 3108 is the rate of the cost function. 3108 in 31 consecutive training epochs approaches a predetermined acceptable cost function.

[0394] Training a neural network combines the processes of forward propagation and backpropagation. Forward propagation is the process by which input data passes through the network from the input layer to the output layer. During forward propagation, the network's weights and biases are used to calculate the output of a given input. Backpropagation, on the other hand, is the process of updating the weights and biases of the network (e.g., a cost function) based on the error. After forward propagation of the IDEP neural network, the output of the network is compared to the true output, and the error is calculated. This error flows back through the entire network from the output layer to the input layer. The weights and biases are adjusted to minimize this error. This process is repeated over multiple iterations or epochs until the network can make accurate predictions.

[0395] The neural network training methods described above are supervised learning, where the network is trained on a labeled dataset (e.g., sample pairs of input user prompts and corresponding output recommendations) and the true output is known. In supervised learning, the network is trained on an unlabeled dataset with the goal of discovering hidden patterns and structures in the data. The network is not provided with a true output, and training is based on the inherent properties of the data. Furthermore, reinforcement learning is a type of learning where an agent learns decision-making from the rewards and punishments it receives based on its actions. Reinforcement learning is usually independent of existing datasets, although some forms of reinforcement learning can utilize a database of past actions, states, and rewards during the learning process. Neural network training methods using labeled datasets fall within the scope of the methods and systems described here, as is evident from the following overview.

[0396] Fig. 32 Based on exemplary examples of the present invention, we provide additional information regarding the training process or machine learning models of IDEP.

[0397] Transformer Model Architecture The Transformer Architecture is a neural network design introduced in the paper "Attention Is All You Need," by Vaswani et al., published in June 2017 (available online). https: / / arxiv.org / abs / 1706.03762 It is incorporated by reference as being fully described in this book. Large-scale Language Models (LLMs) rely heavily on the Transformer Architecture.

[0398] This architecture (see Figure 1 by Vaswani et al.) is based on the concept of "attention," allowing the model to focus on different parts of the input sequence when generating the output. The transformer consists of an encoder and a decoder. The encoder processes the input data, and the decoder generates the output. Each of these components consists of a self-attentional layer and a layer that is fully connected point by point.

[0399] In a trans model, the self-attention layer allows for the evaluation of relationships between different parts of the input sequence when generating the output, capturing long-range dependencies within the data. Meanwhile, fully connected layers are used to transform the output of the self-attention layer, adding complexity and depth to the model's learning ability.

[0400] Transformer models are known for their ability to handle long data sequences and are particularly effective for tasks such as machine translation and text summarization. In the Transformer architecture, positional coding is used to provide the model with information about the relative positions of words in the input sequence. Since the model itself has no sense of order or inherent order, positional coding is a way to inject order information into an order-independent attention mechanism.

[0401] Embedding vector space In the context of neural networks, tokenization refers to the process of converting input and output spaces, such as natural language text or programming code, into discrete units or "tokens." This process makes complex structures easier to manage and converts them into individual elements, allowing models to effectively process and understand the data they can learn from and generate.

[0402] In neural network training, embeddings function as a form of distributed word representation, transforming discrete categorical variables (i.e., tokens) into a continuous vector space (i.e., vector embeddings). This transformation process captures the semantic properties of tokens, allowing tokens with similar meanings to have similar embeddings. These embeddings tightly represent tokens and their semantic relationships. Embeddings are usually represented as vectors, but they can also be represented as matrices or tensors.

[0403] The input to a transformer typically requires a transformation from the input space (e.g., natural language token space) to the embedding space. This process is called "encoding" and converts discrete inputs (tokens) into continuous vector representations (embeddings). This transformation is a prerequisite for the transformer model to process the input data and understand the semantic relationships between tokens (e.g., words). Similarly, the output of a transformer typically requires a transformation from the embedding space to the output space (e.g., natural language tokens, programming code tokens, etc.), which is called "decoding." Therefore, both the training and evaluation (i.e., deployment) of a neural network take place within the embedding domain.

[0404] This document assumes the processes of tokenization, encoding, decoding, and detokenization. In other words, the processes described below occur within an "embedding space." Therefore, while the tokenization and encoding of training data and input prompts may not be explicitly expressed or discussed, they may still be implicitly suggested. Similarly, the decoding and detokenization of neural network outputs may also be implied.

[0405] Training and fine-tuning of machine learning (ML) modules Fig. 32 is an explanatory flow diagram illustrating the various phases and datasets involved in training an IDEP ML model, based on an exemplary example of the present invention.

[0406] The training process begins with step 32 using DE data acquisition, retrieval, assimilation, and generation. In step 3220, the acquired DE data is pre-processed or prepared. In step 3230, the IDEP ML model is trained using the training data. In step 3225, the IDEP ML model is evaluated, validated, and tested, and further improvements are fed back into step 32 for additional training. Once performance reaches an acceptable level, step 3250 selects the optimal IDEP ML parameters.

[0407] The training data 3225 is a dataset containing multiple system input instances (e.g., user input, user prompts, digital twin / physical twin performance data, simulation data, and / or authentication / requirements documents, etc.) and correct results (e.g., updated scripts, DE models, twin configurations, digital twins, physical twins, etc.). The IDEP ML model is trained to optimize performance for a specific target task, such as predicting a specific target output data field within a specific target document, as shown in Figure 32. The training data 3225 may also include a subset for validation and testing of the IDEP ML model as part of the training iterations 3230 and 3240. For NN-based ML models, the quality of the output may depend on (a) the NN architecture design and hyperparameter configuration, (b) NN coefficient or parameter optimization, and (c) the quality of the training dataset. These components can be refined and optimized in various ways. For example, the training data 3225 is extensible through the process of expanding the document database.

[0408] In some cases, additional fine-tuning is performed using 60 phases of fine-tuning data, including 32 iterations of fine-tuning, 3260 steps, and 3270 steps of evaluation, validation, and testing.3255 Fine-tuning in machine learning involves further adjusting ("tuning") parameters of a selected pre-trained model to better suit a particular task or dataset.3255 This technique is particularly useful when dealing with deep learning models trained on large, general training datasets,3225 intended for application to more specialized tasks or smaller datasets.3225 The goal is to refine the model so that it can leverage the knowledge it has already acquired during initial training (often called transfer learning) and perform better on more specific tasks.

[0409] Fine-tuning typically begins with a model already trained on a large benchmark training dataset, such as ImageNet (available at https: / / image-net.org / ) for image recognition tasks. The existing weights learned during the initial training serve as the starting point. During fine-tuning, the model is further trained on a new fine-tuning dataset, which may contain different classes or types of data than the original training set. This additional training phase allows the model to adjust its weights to more accurately capture the features of the new fine-tuning dataset, thus improving performance when optimized for a particular task.

[0410] In some cases, additional testing and validation are performed. 3280-stage DE testing and validation data are performed. 3275. Testing and validation of ML models refers to the process of evaluating the model's performance on separate datasets not used during training. 3275 were performed to ensure generalization to new, unverified data. Validating ML models helps prevent overfitting by ensuring that the model's performance generalizes beyond the training data.

[0411] The validation phase is considered part of ML model development and leads to further fine-tuning, while the test phase is the final evaluation of the model's performance after training and validation. The test phase provides an unbiased evaluation of the final model's performance, reflecting how well the model is expected to perform on unseen data. It is usually conducted after the model has been finalized to ensure an unbiased evaluation.

[0412] When an IDEP ML model is trained, 3230 units are selected, 3250 units are selected, and 3260 units are optionally fine-tuned. Validated and tested, 3280 units are selected, and the process ends with deployment of 32 IDEP ML models. Typically, 32 deployed IDEP ML models receive new DE data, 95 units receive new DE data, and 3285 units receive pre-processed data.

[0413] In machine learning, data preprocessing is tailored to the stage of model development. During model training, preprocessing includes cleaning, normalizing, and transforming raw data into a format suitable for pattern learning. For fine-tuning, preprocessing adjusts the data to match the distribution of a specific target task, ensuring that the pre-trained model effectively transmits knowledge. Validation preprocessing reflects the preprocessing during training, accurately evaluating the model's generalization while avoiding information leaks from the training set. Finally, for deployment, preprocessing often involves dynamic adjustments to ensure that real-world data matches the expectations of the trained model and maintain consistency with the training and validation phases.

[0414] Machine learning algorithms The scope of this invention includes, but is not limited to, a variety of exemplary ML algorithms. Such machine learning algorithms include, but are not limited to, random forests, nearest neighbors, decision trees, support vector machines (SVMs), Adaboost, gradient boosting, Bayesian networks, evolutionary algorithms, and various neural networks (such as deep learning networks (DLNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs)).

[0415] ML modules based on transformers and large-scale language models (LLMs) are particularly well-suited to the tasks described here. The online article "Understanding Large-Scale Language Models – A Transformative Reading List" by S. Raschka (published February 7, 2023, available at https: / / sebastianraschka.com / blog / 2023 / llm-reading-list.html) describes various LLM architectures that fall within the scope of the techniques and systems described in this book and are incorporated by reference as being fully documented in this book.

[0416] The input to each listed ML module is a feature vector containing the input data for that ML module. The output of each ML module is a feature vector containing the output data corresponding to that ML module.

[0417] Before deployment, each of the above ML modules can be trained with one or more sample input datasets and corresponding sample output datasets. Input and output training datasets may be generated from a database containing a history of input instances (e.g., user inputs, user prompts, digital twin / physical twin performance data, simulation data, authentication / requirements documents) and output instances (e.g., updated scripts, DE models, twin configurations, digital twins, physical twins), or they may be synthetically generated by experts.

[0418] Exemplary system architecture Examples of this disclosure include one or more servers (management computing entities), one or more networks, and one or more clients (user computing entities). These components, entities, devices, and systems (similar terms used herein) are cloud-based and may communicate directly or indirectly, for example, over the same wired or different wired or wireless networks. All of these devices, including servers, clients, and other computing entities and nodes, can be operated by the customer themselves (in various architectural configurations, including private clouds), internally by the IDEP provider (in various architectural configurations, including private clouds), and / or on a public cloud.

[0419] Fig. 33 provides an illustrative circuit diagram of a server (management computing entity) 3310 connected via a network 33 for clients (user computing entities) 3033 According to some examples of the present invention, 30 is used for documentation within an interconnected digital engineering platform (IDEP). On the other hand, Fig. 33 shows various system entities as independent entities, and various implementations are not limited to this particular architecture. Furthermore, the terms “client device,” “client computing entity,” “edge device,” and “edge computing system” are equivalent and are used synonymously here.

[0420] Exemplary Management Computing Entity The circuit diagram shown in the figure is provided. 33 For Server or Management Computing Entity 3310. Generally, the terms “computing entity,” “computer,” “entity,” “device,” “system,” and / or similar terms used herein may refer to, for example, one or more cloud servers, computers, computing entities, desktop computers, mobile phones, tablets, phablets, laptops, notebooks, distributed systems, game consoles, watches, glasses, iBeacon, proximity beacon, key fob, radio frequency identification (RFID) tag, earpiece, etc.; scanners, televisions, dongles, cameras, wristbands, wearable items / devices, kiosks, input terminals, servers and server networks, blades, gateways, switches, processing units, processing entities, set-top boxes, relays, routers, network access points, base stations, etc., or any combination of devices or entities adapted to perform the functions, operations, and processes described herein. These functions, operations, and processes include, for example, transmitting, receiving, operating, processing, crawling, displaying, saving, deciding, creating / generating, monitoring, evaluating, and / or comparing (similar terms are used synonymously here). In some implementations, these functions, operations, and processes can be performed on data, content, and information (similar terms used synonymously here) in the same way as they would be used in digital engineering processes.

[0421] One example is a management computing entity 3310, which may be equipped with one or more communication interfaces 3312 for communicating with various computing entities by exchanging, transmitting, receiving, manipulating, processing, displaying, and storing data, content, and information (similar terms used here synonymously). For example, a management computing entity 3310 can communicate with one or more client computing devices 3330, or various other computing entities. The network or communication interface 3312 may support a variety of wired data transmission protocols, such as Fiber Distributed Data Interface (FDDI), Digital Subscriber Line (DSL), Ethernet, Asynchronous Transfer Mode (ATM), Frame Relay, and Data Over Cable Service Interface Specification (DOCSIS). Furthermore, the management computing entity 3310 is capable of wireless communication with external networks and can use a variety of standards and protocols, including General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), CDMA2000 1X (1xRTT), Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Evolutionary Universal Terrestrial Radio Access Network (E-UTRAN), Evolution Data Optimization (EVDO), High Speed ​​Packet Access (HSPA), High Speed ​​Downlink Packet Access (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX), Ultra Wideband (UWB), Infrared (IR) protocol, Near Field Communication (NFC) protocol, Wibree, Bluetooth protocol, Wireless Universal Serial Bus (USB) protocol, and / or other wireless protocols.

[0422] As shown in the figure, 33, as an example, a management computing entity 3310 may contain or communicate with one or more processors 3314 (also called a processor, processing circuit, processing element, or similar terms used here synonymously) which communicate with other elements within the management computing entity 33, for example, 10 people on a bus. As understood, a processor 3314 can be embodied in a variety of forms. For example, a processor 3314 may be implemented as one or more complex programmable logic devices (CPLDs), microprocessors, multicore processors, coprocessing entities, application-specific instruction set processors (ASIPs), graphical processing units (GPUs), microcontrollers, and / or controllers. The term circuit may refer to an entire hardware implementation or a combination of hardware and computer program products. Thus, a processor 3314 may be implemented as an integrated circuit (IC), a special-purpose integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic array (PLA), a hardware accelerator, or other circuit. Therefore, the processor 3314 can be configured to be assigned to a specific purpose or to execute instructions stored on volatile or non-volatile (or non-temporary) media 3316 and 3318, or other configurations 3314 accessible to the processor. Thus, a processor 3314, consisting of hardware products, computer program products, or a combination thereof, may, if properly configured, perform steps and operations in accordance with the examples of this disclosure.

[0423] One example is a management computing entity, which may further include or communicate with non-transient memory,3318 (also known as non-volatile media, non-volatile memory, non-transient memory, physical storage medium, memory, memory storage, or memory circuitry; similar terms are used synonymously here). In some implementations, non-transient memory or storage may include, but not limited to, temporary memory or storage media such as hard disks, ROMs, PROMs, EPROMs, EEPROMs, flash memory, MMCs, SD memory, memory sticks, CBRAMs, PRAMs, FeRAMs, NVRAMs, MRAMs, RRAMs, SONOS, FJG RAMs, millipede memory, and racetrack memory. As will be recognized hereafter, non-volatile (or non-transient) storage or memory media may store cloud storage buckets, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreter code, machine code, execution instructions, etc. A database, database instance, or database management system (as used synonymously here) may refer to a collection of records or data stored on a computer-readable storage medium using one or more database models, such as a hierarchical database model, network model, relational model, entity-relationship model, object model, document model, semantic model, graph model, etc.

[0424] One example is a management computing entity, which may further include or communicate volatile memory (also called volatile memory, memory, memory storage, memory, or circuitry; similar terms are used synonymously here). In some examples, volatile memory devices or memory may include, but are not limited to, one or more volatile memory devices or memory media, such as RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, register memory, etc. As will be recognized in the future, volatile storage media may be used to store databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreter code, machine language, executable instructions, or at least a portion of what is executed by a processor, for example.3314 Therefore, cloud storage buckets, databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, bytecode, compiled code, interpreter code, machine language, executable instructions, etc., may be used to control certain aspects of the operation of managed computing entities,3310 with the help of a processor,3314 and an operating system.

[0425] Although not displayed, the management computing entity 3310 may include or communicate with one or more input elements, such as keyboard input, mouse input, touchscreen / display input, motion input, movement input, audio input, pointing device input, joystick input, or keypad input. The management computing entity 3310 may also include or communicate with one or more output elements that are not displayed, such as audio output, visual output, screen / display output, motion output, or spatial computing output (e.g., virtual reality or augmented reality).

[0426] As you understand, one or more components of managed computing 3310 may be deployed remotely from other managed computing entity components, such as in a distributed system. Furthermore, one or more components may be combined, or additional components that perform the functions described here may be included in the managed computing entity 3310. Thus, the managed computing entity 3310 can be adapted to various needs and situations. As you know, these architectures and descriptions are provided for illustrative purposes only and are not limited to various implementations.

[0427] Exemplary User Computing Entity Users may be human individuals, companies, organizations, departments within organizations, representatives of organizations, algorithms, artificial intelligence, or other software that performs interfaces, or similar artificial users. 33 Furthermore, a graphical schematic representation of the client user computing entity 3330 may be used in conjunction with the embodiment of this disclosure. In various forms, the computing device 3330 may be a general-purpose computing device with dedicated modules for performing digital engineering-related tasks. Alternatively, it may be implemented on the cloud and in a logically and / or physically distributed architecture.

[0428] As shown in the figure, the user computing entity 3330 may include a power supply 3331, an antenna 3370, a radio transmitter 3332, a network and communication interface 3334, and a processor unit 3340 that sends and receives signals to the network and communication interface. The signals provided and received may include signaling information compliant with the applicable air interface standards of the radio system. In this regard, the user computing entity 3330 may be capable of operating with one or more air interface standards, communication protocols, modulation types, and access types. In particular, the user computing entity 3330 may operate according to any of several radio communication standards and protocols, such as those relating to the management computing entity 3310. Similarly, the user computing entity 3330 may operate according to several wired communication standards and protocols, such as those relating to the management computing entity 3310.

[0429] Through these communication standards and protocols, user computing entity 3330 can communicate with various other entities using concepts such as unstructured supplemental service data (USSD), short message service (SMS), multimedia messaging service (MMS), dual-tone multi-frequency signaling (DTMF), and / or subscriber identification module dialer (SIM dialer). User computing entity 3330 can also download firmware, software (including executable instructions, applications, and program modules), operating system changes, add-ons, and updates.

[0430] In some implementations, the processing unit 3340 may be embodied in several different forms. For example, the processing unit 3340 may be implemented as one or more complex programmable logic devices (CPLDs), microprocessors, multicore processors, coprocessing entities, application-specific instruction set processors (ASIPs), graphical processing units (GPUs), microcontrollers, and / or controllers. Furthermore, the processing unit 3340 may be implemented as one or more other processing units or circuits. The term "circuit" may refer to a purely hardware implementation or a combination of hardware and computer program products. Thus, the processing unit 3340 may be implemented as an integrated...

Claims

1. One or more non-temporary physical storage media for storing program code, wherein the program code is executable by a processor, and when the program code is executed by the processor, it causes the processor to execute a computerization process for interacting with live digital objects, and the program code Receiving the live digital object, wherein the live digital object includes digital artifacts extracted from a digital model file via a model representation, and the model representation includes a model type-specific locator for the digital model data, Initiating a connection to a multimodal interface, wherein the multimodal interface is configured to receive input from at least two different modalities, and the multimodal interface is configured to output at least two different modalities, the at least two different modalities including interactive modalities and spatial modalities, Receiving the security level of the first user, Based on the security level of the first user, the first user's access rights to access the digital artifact and the first user's modification rights to modify the digital artifact are determined. Based on the access rights of the first user to access the digital artifact, the digital artifact is output to the multimodal interface through the connection. The multimodal interface receives, through the connection, dialogue input and spatial input from the first user related to the digital artifact, Based on the first user's modification authority to modify the digital artifact, and based on at least one of the interactive input and the spatial input, a modified digital artifact is generated from the digital artifact via the model representation. One or more non-temporary physical storage media containing code for performing the following.

2. The aforementioned program code is: Based on the security level of the first user, display a number of permitted artifacts, Receiving the spatial input from the first user, which includes a first gesture input for selecting one of the plurality of permitted artifacts, Receiving the spatial input from the first user, which includes a second gesture input for positioning one of the plurality of permitted artifacts, Updating the live digital object based on the placement of one of the multiple permitted artifacts, One or more non-temporary physical storage media according to claim 1, further comprising code for performing the following:

3. The aforementioned program code is: The code further includes code for generating an orchestration script that accesses one of the aforementioned permitted artifacts, The aforementioned orchestration script includes instructions for generating new live digital objects, The new security level of the new live digital object is based on a given security level of one of the plurality of permitted artifacts, one or more non-temporary physical storage media according to claim 2.

4. The one or more non-temporary physical storage media according to claim 1, wherein the digital model file is selected from the group consisting of digital engineering (DE) model files, medical model files, supply chain logistics model files, manufacturing model files, and financial model files.

5. One or more non-temporary physical storage media according to claim 1, wherein the model representation includes a model splice connected to the digital model file, the model splice includes one or more splice data items, one or more splice data structures, and a splice function that provides access to the digital artifact, the access to the digital artifact is provided by an item selected from the group consisting of application programming interface (API) endpoints and software development kit (SDK) endpoints.

6. The live digital object is maintained using a software code-defined digital thread, the software code-defined digital thread includes instructions for accessing the model splice, and the software code-defined digital thread includes instructions for accessing the digital model file and different digital model files, one or more non-temporary physical storage media according to claim 5.

7. The one or more non-temporary physical storage media according to claim 6, wherein the digital model file and the different digital model files are from different software tools, and the software code-defined digital threads access different digital artifacts through different model splices connected to the different digital model files.

8. The aforementioned program code is: One or more non-temporary physical storage media according to claim 5, further comprising code for sharing the model splice associated with the modified digital artifact.

9. The program code further includes code for modifying the model representation, the code for modifying the model representation includes code for updating the model splice, and the code for updating the model splice includes code for updating a splice function of the model splice selected from a group of splice functions consisting of a given splice data item of the model splice, a given splice data structure of the model splice, and a given splice function of the model splice, according to claim 5, one or more non-temporary physical storage media.

10. The modified digital artifact appears on the live digital object within a predetermined delay in one or more non-temporary physical storage media according to claim 1.

11. The live digital object is a live digital document, the live digital document includes one or more digital artifacts obtained from one or more model files within a predetermined delay, one or more non-temporary physical storage media according to claim 10.

12. The live digital object is a live digital board, the live digital board displays one or more documents and one or more applications on a 2D screen rendered in a modality of the multimodal interface selected from the group consisting of a 2D display, a 2.5D display, and a 3D semi-immersive or fully immersive display, one or more non-temporary physical storage media according to claim 1.

13. The live digital object is a live digital space, and the live digital space displays one or more documents and one or more other applications in a virtual space within a 3D spatial display, one or more non-temporary physical storage media according to claim 1.

14. The live digital object is stored and accessed from an interconnected digital model platform (IDMP) in one or more non-temporary physical storage media according to claim 1.

15. The aforementioned program code is: Sending the corrected digital artifact to a second user, Receiving a feedback command from the second user, In response to the actions or approvals of the first user, and based on the feedback instructions from the second user, the digital model file is modified. One or more non-temporary physical storage media according to claim 1, further comprising code for performing the following:

16. The said live digital object further includes a second digital artifact accessed via a second model representation, and the program code is, Receiving a second user input from the second user relating to the live digital object, wherein the second user input is an input in a modality selected from the group consisting of the interactive modality and the spatial modality, Based on the second user input, the second digital artifact is further modified to generate a second modified digital artifact. One or more non-temporary physical storage media according to claim 15, further comprising code for performing the following:

17. The live digital object is an object in an interconnected digital model platform (IDMP), the IDP includes a natural language processing (NLP) module configured to interact with the first user based on one or more user inputs, and the multimodal interface is further configured to communicate with the first user in natural language, one or more non-temporary physical storage media according to claim 1.

18. The multimodal interface includes an interactive interface configured to receive audio-based input, wherein at least one of the one or more user inputs includes a voice-based input, one or more non-temporary physical storage media according to claim 17.

19. The multimodal interface includes an interactive interface configured to receive text-based input, and at least one of the one or more user inputs includes the text-based input, one or more non-temporary physical storage media according to claim 17.

20. The spatial input comprises at least one of augmented reality (AR) input, virtual reality (VR) input, mixed reality (MR) input, and gesture input, for one or more non-temporary physical storage media according to claim 1.

21. The multimodal interface includes a spatial computing interface comprising a video input module, an audio input module, and a gesture input module, wherein video-based input is acquired via video received by the video input module, audio-based input is acquired via audio received by the audio input module, and at least one gesture is extracted via the gesture input module, one or more non-temporary physical storage media according to claim 20.

22. The spatial computing interface is further configured to receive contextual information associated with the video-based input, and the one or more non-temporary physical storage media are as described in claim 21.

23. A system for interacting with live digital objects, At least one processor, At least one memory for storing program code, wherein the program code is executable by the at least one processor to cause the at least one processor to execute a process for interacting with the live digital object, and the program code is Receiving the live digital object, wherein the live digital object includes digital artifacts extracted from a digital model file via a model representation, and the model representation includes a model type-specific locator for the digital model data, Initiating a connection to a multimodal interface, wherein the multimodal interface is configured to receive input from at least two different modalities, and the multimodal interface is configured to output at least two different modalities, the at least two different modalities including interactive modalities and spatial modalities, Receiving the security level of the first user, Based on the security level of the first user, the first user's access rights to access the digital artifact and the first user's modification rights to modify the digital artifact are determined. Based on the access rights of the first user to access the digital artifact, the digital artifact is output to the multimodal interface through the connection. The multimodal interface receives, through the connection, dialogue input and spatial input from the first user related to the digital artifact, Based on the first user's modification authority to modify the digital artifact, and based on at least one of the interactive input and the spatial input, a modified digital artifact is generated from the digital artifact via the model representation. The at least one memory containing code for performing the following: The system including the above.

24. A computer-based method for interacting with live digital objects, Receiving the live digital object, wherein the live digital object includes digital artifacts extracted from a digital model file via a model representation, and the model representation includes a model type-specific locator for the digital model data, Initiating a connection to a multimodal interface, wherein the multimodal interface is configured to receive input from at least two different modalities, and the multimodal interface is configured to output at least two different modalities, the at least two different modalities including interactive modalities and spatial modalities, Receiving the security level of the first user, Based on the security level of the first user, the first user's access rights to access the digital artifact and the first user's modification rights to modify the digital artifact are determined. Based on the access rights of the first user to access the digital artifact, the digital artifact is output to the multimodal interface through the connection. The multimodal interface receives, through the connection, dialogue input and spatial input from the first user related to the digital artifact, Based on the first user's modification authority to modify the digital artifact, and based on at least one of the interactive input and the spatial input, a modified digital artifact is generated from the digital artifact via the model representation. A method performed on the computer, including the following:

Images