Application execution systems, methods for modifying applications, computing devices, and programs

Abstract

We provide an application execution system that enables dynamic changes to running applications. [Solution] The system includes an application definition model that maintains GUI configuration information, driver configuration information that abstracts access to functions or resources used by logic configuration information, and logic configuration information that defines the behavior of the application by referring to at least one of the GUI configuration information and driver configuration information as an integrated application state having mutual reference relationships; a runtime environment that executes the integrated application state without compiling or building; and an integrated update control unit that receives change commands from an AI agent for at least two of the GUI configuration information, driver configuration information, and logic configuration information, and updates the integrated application state by applying the change commands while maintaining consistency between the GUI configuration information, driver configuration information, and logic configuration information.

Description

Technical Field

[0001] The present invention relates to an application execution system that integrally manages the configuration of a user interface (GUI), the configuration of a driver that abstracts access to functions and resources used by logic, and the configuration of logic that references these, and particularly relates to a technology that realizes dynamic modification of an application in cooperation with an AI agent.

Background Art

[0002] Conventionally, as a technology related to GUI controller design and remote control, a system is known that designs a GUI controller displayed on a touch panel display and accepts operation inputs for operating a target device via the GUI controller (see Patent Document 1). The system described in Patent Document 1 includes an editing function for a controller set composed of controller parts, an editing function for user programs, and an editing function for drivers, and enables updating of the controller set and user programs while an engine activates the execution environment of the user programs, thereby realizing real-time control modification.

[0003] The technology described in Patent Document 1 integrates and manages three configuration layers: a GUI (controller set), a driver, and a user program (logic), enabling real-time application of updates to these layers while the engine's execution environment is running. However, Patent Document 1 assumes that the user manually edits each component, and does not disclose a technology in which an AI agent understands the state of the running application and autonomously generates and applies changes. Furthermore, the GUI configuration information described in Patent Document 1 is limited to controller parts (joysticks, buttons, sliders, etc.) intended for remote control of hardware devices, and is not configured as general-purpose GUI configuration information that includes user interface elements (HTML elements, CSS style definitions, etc.) in web applications and cloud services. Moreover, Patent Document 1 neither discloses nor suggests a technology for abstracting and providing dynamic state information of the running runtime environment in a format that an AI agent can process.

[0004] In recent years, code generation technologies and application development support tools utilizing large-scale language models (LLMs) have been rapidly developing. These technologies contribute to the efficiency of software development by generating source code from natural language instructions or by making tool calls based on API schemas.

[0005] However, conventional LLM-based code generation technologies primarily focus on generating source code, and applying the generated code involves a compilation or build process, making them unsuitable for real-time application of changes to a running application. Furthermore, conventional LLM tool usage (function calling) relies on statically defined API schemas as input, and no technology has been established to utilize the dynamic state of a running application as input.

[0006] In recent years, large-scale language model function calling protocols and agent orchestration techniques have been proposed. These techniques provide a foundation for large-scale language models to autonomously call and process external tools or services. However, they all presuppose a statically defined toolset and do not disclose mechanisms for dynamically exposing the dynamic state of a running application as such tool definitions, nor for applying changes while maintaining referential integrity of configuration information across multiple configuration layers.

[0007] Furthermore, when an AI agent simultaneously applies changes across multiple layers to an integrated application state where the three constituent layers—GUI, driver, and logic—have mutually referential relationships, technical challenges arise that do not occur with single-layer changes. Specifically, challenges such as maintaining referential integrity between layers, controlling the dependency propagation of changes, the need for atomic application, and non-destructive updating of the execution context have not been resolved by conventional hot-reload or AI code generation technologies. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] U.S. Patent No. 11,249,732 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] The technology described in Patent Document 1 integrates and manages three configuration layers: a GUI (controller set), a driver, and a user program (logic), enabling real-time updates while the execution environment is running. However, it assumes that the user manually edits each component, and does not envision a technology in which an AI agent autonomously generates and applies changes. Furthermore, the GUI configuration information is limited to controller parts for hardware control and does not include general-purpose UI elements, including web elements. Moreover, a mechanism for abstracting and providing the runtime state in a format that can be processed by the AI ​​is not disclosed. In the present invention, all operations on the runtime environment are unified as a boundary representation using serializable event data objects, thereby handling user manual editing, AI autonomous modification, and remote operation on the same platform, which is also fundamentally different from Patent Document 1.

[0010] In the system described in Patent Document 1, the user manually edits each component, performing operations one by one while confirming the consistency of the edited content. In contrast, when an AI agent is the primary modifier, the following new technical challenges arise that do not occur with manual editing.

[0011] (1) Challenges of referential integrity associated with simultaneous multi-layer changes: An AI agent can generate large-scale changes spanning multiple configuration layers at once in response to a single instruction. For example, it can generate the addition of a user interface element, the addition of a driver corresponding to that element, and the addition of logic to control the driver in response to the operation of that element, all from a single instruction. In this case, a dependency on the order of operations may arise, where one component references an uncreated component in another configuration layer. In manual editing, this is a problem that the user can avoid by being aware of the order of operations, but in autonomous change generation by an AI agent, automatic constraints on the order of operations or ex post consistency verification are necessary.

[0012] (2) Issues with residual references following the deletion of components: When an AI agent deletes a user interface element or driver instance, references (identifiers) to the deleted component remain in the logic configuration information, which can cause runtime errors. In manual editing, the user also modifies the related logic, but the AI ​​agent needs to fully understand how the component to be deleted is referenced by other configuration layers. To understand this, a mechanism (dependency graph) for analyzing the reference relationships between configuration layers is essential.

[0013] (3) Safety challenges due to the unpredictability of changes: With manual editing, changes are generally predictable because the user selects and edits from known user interface elements and drivers. On the other hand, because AI agents generate changes based on the output of large-scale language models, they may generate logic containing infinite loops, runaway control signals via drivers, or changes that exceed predetermined resource consumption limits. In particular, in environments where hardware devices are controlled via driver configuration information, incorrect changes by AI agents can cause physical damage, making safety verification before applying changes essential.

[0014] (4) Challenges of state accumulation associated with iterative changes: The AI ​​agent may execute an iterative improvement loop that proposes additional changes after reviewing the results of the changes. In this iterative process, each change cumulatively alters the state of the integrated application, resulting in a state accumulation where a change depends on the results of past changes, and past changes become prerequisites for future changes. If inconsistencies are detected during the change process, transaction control (snapshot creation and rollback) is required to restore the integrated application state to a consistent state.

[0015] In addition, for an AI agent to understand the state of a running application and generate appropriate changes, a technology is needed to convert the state information of the runtime environment into a format that can be processed by a large-scale language model. However, sending the entire state of the runtime environment directly to a large-scale language model is impractical due to the constraints of the number of tokens, and there is also the challenge that a large amount of context containing information irrelevant to the changes will reduce the accuracy of the AI ​​agent's decisions.

[0016] The present invention was made to solve the above problems and aims to provide an application execution system, method, apparatus, and program that maintains GUI configuration information, driver configuration information, and logic configuration information as an integrated application state, and enables dynamic changes to a running application in cooperation with an AI agent through non-destructive updates of the execution context. [Means for solving the problem]

[0017] As a result of investigating the above problems, the inventors of this invention have found that by integrating the components of an application into a single data structure and treating changes to that data structure as a consistent operation, dynamic changes can be achieved while maintaining consistency between layers. Furthermore, in this invention, the integrated application state is maintained as a serialized representation having the same structure as the execution representation of the runtime environment. This allows the execution state and the state manipulated from the outside to be treated in the same format, and changes to the state can be applied directly. As a result, change generation and application by the AI ​​agent can be achieved without additional transformation processing.

[0018] In other words, based on the above principle, the application execution system according to the first aspect of the present invention is an application execution system that operates on a computing device and comprises an application definition model that holds GUI configuration information defining user interface elements, driver configuration information abstracting access to functions or resources used by logic configuration information, and logic configuration information defining the behavior of the application by referencing at least one of the GUI configuration information and the driver configuration information as an integrated application state having mutual reference relationships; a runtime environment that interprets and executes the integrated application state without going through a compilation or build process; an integrated update control unit that receives a change instruction from an AI agent that includes changes to at least two of the GUI configuration information, driver configuration information, and logic configuration information, and updates the integrated application state by applying the change instruction while maintaining the consistency of the reference relationships while tracking the reference relationships between the GUI configuration information, driver configuration information, and logic configuration information included in the integrated application state; and an AI agent interface that presents an abstracted world state generated from the integrated application state to the AI ​​agent, receives change instructions generated by the AI ​​agent and transmits them to the integrated update control unit, wherein the integrated update control unit updates the integrated application state while maintaining the execution state of the runtime environment.

[0019] A second aspect of the present invention relates to a method for modifying an application in a runtime environment in which the integrated application state described above is being executed, in cooperation with an AI agent, comprising the steps of: receiving instructions from a user regarding the modification of the application; generating an abstract world state by extracting configuration information related to the instructions from information contained in the integrated application state and the execution state of the runtime environment to reduce the amount of information; transmitting the instructions and the abstract world state to a large-scale language model; receiving a modification instruction from the large-scale language model, which includes modifications to at least two of GUI configuration information, driver configuration information, and logic configuration information; and updating the integrated application state while maintaining the execution state of the runtime environment by applying the modification instruction while tracking the reference relationships between the GUI configuration information, driver configuration information, and logic configuration information contained in the integrated application state and maintaining the consistency of said reference relationships.

[0020] A computing device according to a third aspect of the present invention comprises a touch-sensitive display, a processor, and memory, and realizes the functions of the above-mentioned application execution system within a single application.

[0021] A program according to a fourth aspect of the present invention is a program for causing a computer to function as the above-mentioned application execution system. [Effects of the Invention]

[0022] According to the present invention, GUI configuration information, driver configuration information, and logic configuration information are held as an integrated application state, and change commands received from the AI ​​agent are applied while maintaining referential integrity, thereby enabling dynamic changes to the application based on the user's natural language instructions without disrupting the consistency between layers.

[0023] Furthermore, even when the aforementioned change command is an integrated change command spanning multiple configuration layers, by selectively applying it only to the specified configuration information in the integrated application state, the application can be updated without interrupting running processes, discarding execution contexts, or disconnecting established communication connections.

[0024] Accordingly, (i) maintenance of reference consistency for changes spanning multiple layers, (ii) selective update while maintaining the execution state, and (iii) provision of an abstracted world state that can be handled by an AI agent are mutually interlocked, whereby continuous operation and iterative improvement are both achieved particularly in real-time applications involving hardware control and Web applications in operation as cloud services.

[0025] Furthermore, by generating an abstracted world state that deterministically extracts and compresses only relevant parts based on a dependency graph from the running state of the runtime environment, it is possible to efficiently provide information that is necessary and sufficient for the AI agent to make change determinations, without relying on heuristic full-text search, while complying with the input constraints of large language models.

[0026] The above effects stem from the fact that the runtime environment of the present invention is configured as an execution platform optimized for autonomous modification operations by AI agents. Specifically, these include: (1) the integrated application state is maintained as a text-based serialized representation that is directly interpretable by large language models, so that the AI agent can read the structure and state of the running application; (2) the self-describing part type definition allows the AI agent to dynamically recognize available operation targets and their parameter structures; and (3) the reference structure based on identifiers between configuration layers enables the AI agent to track the impact of a modification to one component on other configuration layers. In conventional source code-based applications, it is difficult to obtain such information from execution binaries that have undergone compilation or build processes, and the runtime environment based on the integrated application state of the present invention has an essential advantage in cooperative operation with AI agents.

[0027] The identity of the above representations provides the following technical effects. First, since the processing of generating the abstract world state is implemented as filtering and summarization processing on the serialized representation of the integrated application state, no information loss occurs when converting the internal state of the running runtime environment to an external representation. Second, modification instructions generated by the AI agent can be directly applied as addition of components, modification of attribute values, or modification of reference relationships in the serialized representation of the integrated application state, and no additional analysis steps (such as source code parsing, compilation, and linking) are required to convert the output of the AI agent into the internal representation of the runtime environment. Third, by re-serializing the state of the runtime environment after the modification is applied, the difference between the states before and after the modification can be compared on the same data format, which serves as a basis for convergence determination in the iterative improvement loop. Brief Description of the Drawings

[0028] [Figure 1] This is a block diagram showing the overall configuration of an application execution system according to an embodiment of the present invention. [Figure 2] This is a conceptual diagram showing the data structure of the integrated application state. [Figure 3] This figure shows an example of a specific JSON structure (controller part type) for the state of an integrated application. [Figure 4] This diagram shows an example of a specific JSON structure (Web / Document element type) for an integrated application state. [Figure 5] This diagram shows the structure of layout management using a scene graph. [Figure 6] This figure shows an abstract syntax tree representation of the logic configuration information. [Figure 7] This sequence diagram shows the processing flow of an application change pipeline in conjunction with an AI agent in the unified change instruction form (Form A). [Figure 8] This sequence diagram shows the processing flow of an application change pipeline in conjunction with an AI agent in the sequential form (Form B) of an integrated change instruction. [Figure 9] This is a flowchart showing the process of generating an abstracted world state. [Figure 10] This is a conceptual diagram illustrating the construction of a dependency graph and the process of extracting related nodes. [Figure 11] This figure shows a comparison of the number of tokens before and after generating an abstracted world state. [Figure 12] This is a flowchart showing the atomic application process for integrated change instructions. [Figure 13] This is a state transition diagram showing a series of transitions in transaction control during non-destructive execution context updates. [Figure 14] This is a timing diagram illustrating the operation of thread consistency guarantee. [Figure 15] This is a flowchart showing the chained update process for referential integrity. [Figure 16] This diagram shows the structure of the mapping table between configuration information IDs and memory addresses. [Figure 17] This flowchart shows the operation of the iterative improvement loop and convergence control. [Figure 18] This is a block diagram of the standalone device configuration. [Figure 19] This is a block diagram of the multi-platform deployment configuration. [Figure 20] This is a conceptual diagram illustrating the integration changes to the web dashboard application. [Modes for carrying out the invention]

[0029] Embodiments of the present invention will be described in detail below with reference to the drawings. Note that the following embodiments are not limiting to the present invention, and various modifications are possible within the scope of the technical concept of the present invention.

[0030] (Definition of terms) To facilitate understanding of this embodiment, key terms are defined below. In this specification, the following terms have the meanings set forth below.

[0031] In this embodiment, the most important configuration is the "integrated application state." The integrated application state refers to a single logical data structure in which GUI configuration information, driver configuration information, and logic configuration information are interconnected through references using instance identifiers. The three configuration layers share reference relationships within a single data structure and are subject to modification, verification, and execution as a whole. A key feature of this invention is that this integrated application state functions as the target of autonomous modification by the AI ​​agent.

[0032] A "modification instruction" is an instruction that instructs the addition, modification, or deletion of at least one piece of configuration information included in the integrated application state. A modification instruction may be an instruction that includes a modification to only one configuration layer, or an instruction that includes a modification spanning multiple configuration layers.

[0033] An "integrated change instruction" refers to an instruction among the aforementioned change instructions that includes changes spanning at least two or more configuration layers among GUI configuration information, driver configuration information, and logic configuration information. An integrated change instruction is implemented in one of the following two forms.

[0034] (Form A) Batch Form: A form in which a combination of changes spanning multiple configuration layers is included as a single instruction in an indivisible manner and applied atomically. The large-scale language model generates a JSON structure containing all changes in a single response, and the integrated update control unit 130 applies the structure in a batch.

[0035] (Form B) Sequential Form: In this form, a large-scale language model sequentially issues multiple individual function calls, determining the next operation while checking the result of each function call, thereby achieving changes that span multiple structural layers. Each function call may be an operation on a single structural layer, but the entire series of function calls constitutes an integrated change instruction. In the sequential form, consistency verification is performed after the completion of the series of function calls to ensure referential consistency between layers.

[0036] In either form, it is guaranteed that inter-layer referential integrity in the integrated application state is maintained after the application of the integrated change instruction.

[0037] "Abstracted world state" refers to a state representation suitable for input to a large-scale language model, extracted and compressed from the entire state information of the runtime environment through dependency graph-based relevance determination. Abstracted world state includes one or both of the following two forms:

[0038] (Form I) Tool Definition Format: A representation that structures existing components included in the integrated application state as tool definitions in the tool invocation protocol of the large-scale language model. The instance identifier (instanceId) of each component, the current attribute value, and the operations that can be performed on that component are described as the parameter schema of the tool definition. In the sequential form (Form B), the large-scale language model can directly recognize the current component as an operation target. This allows for dynamic tool definition of operation targets in accordance with updates to the integrated application state, rather than being limited to statically fixed tool definitions.

[0039] (Form II) Serialized Summary Format: A text-based serialized representation of the integrated application state, with information reduced at a level of abstraction corresponding to the type of configuration information. Specifically, information is reduced by simplifying position and size information for GUI configuration information, omitting connection parameters for driver configuration information, and converting logic configuration information into a summarized representation of AST node sequences. In the Batch Form (Form A), this enables a large-scale language model to generate modification instructions after understanding the overall structure of the integrated application state as context.

[0040] The generation of the abstracted world state described above is possible because the integrated application state of the present invention has the same representation between the runtime environment's execution representation and the serialized representation. That is, the serialized text representation of the integrated application state includes instances of GUI configuration information and their attribute values, instances of driver configuration information and their connection status, AST node groups of logic configuration information and their connection relationships, and the current values ​​of variables during execution. By loading this serialized representation into the runtime environment, the application can be immediately restored to its original execution state without going through the compilation or build process. In other words, the internal representation executed by the runtime environment, the external representation read and written by the AI ​​agent, and the saved representation stored for persistence are represented by structurally identical data formats. In conventional source code-based applications, the source code (representation read and written by the developer), the compiled binary (representation executed by the runtime), and the memory state during execution (internal state of the process) are all different representations, and conversion between them involves irreversible information loss. The integrated application state of the present invention eliminates this separation of representations, and by having a single data format fulfill all three roles, it provides a foundation for realizing a closed loop in which an AI agent can directly read and modify the state of a running application and immediately observe the results of those modifications as execution results.

[0041] "User interface elements" refer to a general term for visual or manipulative components used in interaction with the user. This is not limited to controller parts for hardware control (joysticks, buttons, sliders, etc.), but also includes HTML elements (div, form, input, canvas, etc.), CSS style definitions, SVG elements, Web components, and user interfaces composed of combinations thereof.

[0042] An "AI agent" is an autonomous software entity based on a Large-Scale Language Model (LLM) that generates modification commands for the integrated application state in response to user instructions. The AI ​​agent receives an abstracted world state generated from the execution state of the runtime environment as input and generates output that instructs the addition, modification, or deletion of at least one of the following: GUI configuration information, driver configuration information, and logic configuration information. The output of the AI ​​agent is obtained in either the form of a single JSON structure (batch form) or a series of tool calls (sequential form). In this specification, the AI ​​agent includes both those provided via API as a cloud service and those based on a Small-Scale Language Model (SLM) that run locally.

[0043] (Overall system configuration) Figure 1 is a block diagram showing the overall configuration of the application execution system 100 according to this embodiment. The application execution system 100 comprises a runtime environment 120, an integrated update control unit 130, an AI agent interface 140, and an integrated editor 150. These components cooperate with each other via an application definition model 110. The application definition model 110 is a data structure handled by the application execution system 100, and represents an integrated application state in which GUI configuration information 111, driver configuration information 112, and logic configuration information 113 are interconnected via references using an instance identifier (instanceId). In this embodiment, the integrated application state is represented as a data structure in JSON (JavaScript Object Notation) format, but is not limited to this, and MessagePack, Protocol Buffers, or other serialization formats can also be used.

[0044] The implementation form of the application execution system 100 is not limited to a specific hardware configuration. For example, it can operate as a single application on a smartphone or tablet device, as a desktop application on a personal computer or workstation, in a distributed configuration where the runtime environment 120 is placed on a cloud server and the integrated editor 150 is placed on a client terminal, or in a cloud-native configuration where each component is distributed across multiple server nodes. In any of these configurations, the runtime environment 120, the integrated update control unit 130, and the AI ​​agent interface 140 work together by sharing the integrated application state, which is represented as the application definition model 110.

[0045] The integrated editor 150 provides a user interface that enables editing of GUI configuration information 111, driver configuration information 112, and logic configuration information 113 on a display device equipped with a touch-sensitive display or pointing device. The integrated editor 150 has the function of dynamically generating the editing interface based on part type definitions. User editing operations are sent to the integrated update control unit 130 as serializable event data objects and reflected in the integrated application state.

[0046] The runtime environment 120 interprets and executes the integrated application state immediately as an execution state without going through the compilation or build process. The runtime environment 120 is implemented as a virtual machine (VM) and sequentially interprets and executes the programs defined in the logic configuration information 113. The runtime environment 120 includes a GUI drawing engine 121, a communication management unit 122, and a script execution unit 123. The GUI drawing engine 121 selects one of the drawing backends, native UI drawing, HTML / CSS drawing, or OpenGL / Vulkan drawing, depending on the deployment platform. The communication management unit 122 establishes and maintains connections with external devices and services based on the driver configuration information 112. The script execution unit 123 interprets and executes the AST nodes of the logic configuration information 113 and performs event processing and driver operations. In this embodiment, the integrated editor 150 and the runtime environment 120 operate within a single application on the same computing device.

[0047] The AI ​​agent interface 140 generates an abstracted world state from the running state of the runtime environment 120, sends it to the Large-Scale Language Model (LLM) along with user instructions, interprets the response received from the LLM as a single JSON structure in the case of a batch form of integrated change instructions (form A), or as a sequence of tool calls in the case of a sequential form (form B), and passes it to the integrated update control unit 130. The AI ​​agent interface 140 includes a world state generation unit 141, a dependency graph construction unit 142, a relevance determination unit 143, and an abstraction transformation unit 144. The AI ​​agent interface 140 performs API communication with the LLM on a thread independent of the main thread of the runtime environment 120, and maintains the running state of the runtime environment 120 even during communication.

[0048] The integrated update control unit 130 receives integrated change instructions issued from the AI ​​agent interface 140 or the integrated editor 150 and selectively updates the configuration information specified by the integrated change instructions from the integrated application state. The integrated update control unit 130 includes a reference consistency maintenance function 131, a consistency verification unit 132, a safety verification unit 133, and a transaction control unit 134. After applying the update, the integrated update control unit 130 instructs the runtime environment 120 to reload the changed configuration information, and reflects the integrated application state in the execution state of the runtime environment 120 without interrupting any running processes.

[0049] (Data structure of integrated application state) Figure 2 is a conceptual diagram showing the data structure of the integrated application state. This data structure is designed to uniformly manage the reference relationships between layers. The integrated application state consists of three configuration layers: GUI configuration information 111, driver configuration information 112, and logic configuration information 113. Each configuration layer is represented as a collection of instances, and instances within logic configuration information 113 refer to instances of GUI configuration information 111 and driver configuration information 112 by identifiers (instanceId).

[0050] (Serialized representation of integrated application state) In this embodiment, the integrated application state is maintained as a serializable data structure having the same structure as the execution state of the runtime environment. By using a serializable representation identical to that of the execution state, the integrated application state can be directly accessed and updated from the outside and applied to the runtime environment without any conversion process.

[0051] The integrated application state includes GUI configuration information, driver configuration information, and logic configuration information, and these components form mutually referential relationships through identifiers. This allows the application structure, which spans multiple configuration layers, to be represented uniformly as a single data structure.

[0052] Each component is held as an instance described in serial form, and each instance is assigned a unique identifier (instanceId). References between components are expressed using this identifier.

[0053] For example, the integrated application state is represented in JSON format, and each GUI element, driver instance, and logic component is described as an object containing corresponding attributes and reference information. In this case, the AST node included in the logic configuration information represents the relationships between components by referencing the instanceId of the target GUI element or driver instance.

[0054] This serialization representation ensures that the integrated application state is maintained in a consistent state between its internal representation in the runtime environment and its externally manipulated data representation, making it possible to directly generate and apply changes to that state.

[0055] Specific examples of such reference structures are shown in Figures 3 and 4. Figure 3 shows an example of a specific JSON structure (controller part type) of an integrated application state. As shown in Figure 3, the AST node columns included in the controller part, driver, and runtime state can be related by instanceId references within a single JSON object.

[0056] Figure 4 shows an example of a specific JSON structure (Web / Document element type) of the integrated application state. As shown in Figure 4, instances of Web / Document element types and driver instances such as HTTP clients can be represented with a reference structure similar to that in Figure 3.

[0057] In traditional applications, user interfaces, drivers, and logic are typically maintained in different representational formats and management units, and the relationships between these components are managed in a distributed manner as code, configuration files, or runtime internal state. Therefore, when making changes across these multiple configuration layers, separate update processes and consistency adjustments are required for each layer, making it difficult to maintain the consistency of the changes.

[0058] In contrast, in this embodiment, GUI configuration information, driver configuration information, and logic configuration information are uniformly managed as reference relationships by identifiers within the same serialized representation. Therefore, relationships spanning multiple configuration layers are explicitly represented within a single data structure, and changes can be generated and applied integrally as operations on that structure.

[0059] (Self-descriptive part type definition) The integrated application state includes part type definitions that define attribute schemas for each type of user interface element. Part type definitions are defined for both controller part types in category A and web / document element types in category B.

[0060] A part type definition contains self-descriptive attribute information, including the name, data type, default value, range constraints, candidate options, and dependencies between attributes for each attribute. The integrated editor 150 dynamically generates an editing interface based on this part type definition. This ensures that even when a new UI element type is added, an editing interface suitable for that type is automatically provided without modifying the program code of the integrated editor 150.

[0061] (GUI configuration information) GUI configuration information 111 defines the user interface of the application. GUI configuration information 111 includes a collection of instances of user interface elements, attributes of each instance, and a scene graph that defines the hierarchical arrangement of the instances.

[0062] In the present invention, the user interface elements defined by the GUI configuration information 111 include the following two categories.

[0063] (Category A) Controller Part Type: These are dedicated UI components suitable for remote control of hardware devices, including joysticks, cross keys, buttons, switches, sliders, dials, gesture inputs, text output, video output, etc. Each controller part type has attributes suitable for hardware control, such as the range of operation input values, notification frequency, and coordinate information.

[0064] (Category B) Web / Document Element Types: These are elements that constitute the user interface of web applications and cloud services, and include HTML block elements (div, section, article, header, footer, etc.), HTML form elements (input, select, textarea, button, etc.), HTML display elements (table, list, img, video, etc.), CSS definitions (style rules, animation definitions, media queries, etc.), SVG elements, canvas elements, and custom web components. Each element type has attributes such as HTML attributes, CSS properties, and event handler assignments.

[0065] The user interface elements of categories A and B described above are all represented as instances within the GUI configuration information 111 using the same data structure. That is, they have a common instance representation consisting of an instanceId, element type (partType / elementType), and attribute dictionary, and the method of referencing them from the logic configuration information 113 is also the same. This makes it possible to define applications that mix controller part types and Web / document element types (e.g., a configuration that combines a hardware control panel with a web-based data visualization dashboard) as a single integrated application state.

[0066] (Layout management using scene graphs) The hierarchical arrangement of instances is managed by a scene graph. Figure 5 shows an example of the sceneGraph array in the JSON that serializes the integrated application state, illustrating the structure of layout management by the scene graph. In this embodiment, this example uses a format consistent with the data structure described in the technical documentation provided by the client prior to filing. As shown in Figure 5, a node with type boxLayout corresponds to a layout container that includes automatic placement rules such as position, size, orientation, alignment, margin, and spacing. Its children list child nodes with type parts, and the instanceId of each child node references the corresponding instance in the GUI configuration information 111. Additionally, a parts node placed directly under the root as another element of the array may indicate an instance that is placed outside the scope of the automatic placement rules of the container.

[0067] Integrated Editor 150 includes editing functionality for a scene graph that defines the hierarchical arrangement of user interface elements. The scene graph includes a layout container, which defines automatic placement rules including the orientation, spacing, and margins of child elements. For Web / document element types, the layout container has attributes corresponding to CSS Flexbox or Grid layout.

[0068] As described above, the user interface is defined by instances and scene graphs, but the GUI configuration information 111 may further include multiple layout variants that redefine the position and size of user interface elements according to the screen size and screen orientation. For example, different layout definitions may be associated with conditions such as portrait, landscape, and desktop display.

[0069] Among the aforementioned layout variants, those relating to Web / document element types can be expressed as conditional branches equivalent to CSS media queries. When runtime environment 120 operates on a web browser, these conditional branches can be output as corresponding CSS media queries.

[0070] (Driver configuration information) The driver configuration information 112 abstracts access to the functions and resources used by the logic configuration information 113. The driver configuration information 112 defines multiple driver types, including the following four categories.

[0071] (Category I) Hardware device control driver type: Drivers that control hardware devices via physical communication channels such as BLE (Bluetooth Low Energy), serial communication, GPIO, I2C, SPI, etc.

[0072] (Category II) Driver Types for Software Service Connection: Driver types that connect to cloud services or external software systems via network protocols such as RESTAPI or WebSocket. Not limited to RESTAPI and WebSocket, this includes GraphQL (including schema definitions and type-safe queries), gRPC (method calls based on Protocol Buffers definitions), MQTT (publish-subscribe messaging), database connections (SQL and NoSQL), etc. Users can register driver types that implement new protocols according to the schema of the driver type definition, and thereafter the AI ​​agent can refer to and operate that driver type using the same operation patterns as other existing driver types.

[0073] (Category III) Driver type for providing local libraries: This driver type makes libraries that provide utility functions such as string manipulation (StringUtils, etc.), mathematical operations, date processing, and data conversion accessible as driver instances from the logic configuration information 113. The driver type for providing local libraries provides functions that are completed within the application and do not involve network communication to the logic configuration information.

[0074] (Category IV) Drivers for device resource access: These drivers access hardware resources provided by the device or resources provided by the OS, such as cameras, microphones, GPS, accelerometers, and local file systems, via the OS API. While Category I drivers for hardware device control abstract the communication channel with external devices, Category IV drivers abstract access to resources provided by the device itself.

[0075] As the four categories above indicate, the driver configuration information 112 unifies external contacts of different natures—controlling hardware devices, communicating with cloud services, using local libraries, and accessing device resources—into a single abstract concept called "driver." While this unified abstraction concept shares some similarities with the driver abstraction concept in Patent Document 1, in the present invention it has new technical significance in the context of autonomous modification by an AI agent. In conventional software development, external contacts are managed individually by different means (such as installing libraries via a package manager, calling SDK-specific APIs, and implementing client solutions for each communication protocol), requiring an AI agent to have knowledge and procedures specific to each means to operate them. In contrast, the unified abstraction provided by the driver configuration information allows the AI ​​agent to control BLE devices, call REST APIs, and use library functions using the same operation pattern—"adding a driver instance" and "referencing an instance identifier from logic configuration information"—regardless of the type of external contact.

[0076] Each driver type includes a driver type definition that specifies the interface types it supports. Interface types include communication channels (BLE, HTTP, etc.), local APIs (library function calls, etc.), and OS services (camera API, location API, etc.). A key feature is that users can add new driver types. That is, users can extend the system's capabilities by defining new driver types according to the driver type definition schema and registering implementations of the corresponding interfaces.

[0077] (Logic configuration information) The logic configuration information 113 defines the behavior of the application by referencing instances of user interface elements in the GUI configuration information 111 and driver instances in the driver configuration information 112 using identifiers. In this embodiment, the logic configuration information 113 is represented as an Abstract Syntax Tree (AST) composed of interconnected AST nodes. Figure 6 shows an example of a chained representation of AST nodes in the logic configuration information 113. As shown in Figure 6, nodes such as event initiation, value acquisition, and driver transmission are linked by next references, and instances of GUI configuration information or driver configuration information can be referenced by instanceId.

[0078] Each AST node has an opcode and parameters. The connection relationships between AST nodes are expressed as next references and parent references. Here, the next reference indicates the next node to which the node should transition after processing is complete, and defines the execution order. On the other hand, the parent reference indicates the higher-level node to which the node belongs, and defines the logical parent-child relationship that restores control in a branching or nested structure. In particular, AST nodes that reference an instance of GUI configuration information 111 or an instance of driver configuration information 112 hold an instance identifier by the instanceId attribute and form a reference relationship between layers. In this embodiment, AST nodes are automatically constructed when user interface elements (parts) are generated. That is, during the part generation process, the event-starting AST node and value-acquisition AST node corresponding to the part are automatically defined in a form that references the part's instanceId and registered with the script execution unit 123 of the runtime environment 120.

[0079] (AI agent collaboration pipeline) Figure 7 is a sequence diagram showing the processing flow of an application change pipeline in conjunction with an AI agent in the batch form of an integrated change instruction (Form A). Figure 8 is a sequence diagram showing the processing flow of the same pipeline in the sequential form of an integrated change instruction (Form B). The AI ​​agent-cooperative pipeline shows the process from receiving user instructions to updating the integrated application state, and steps (1) to (3) described below are common to both Form A and Form B. Steps (4) and (5) are distinguished depending on whether the change is implemented by Form A or Form B of the integrated change instruction. Figure 7 mainly shows the message flow corresponding to Form A (receiving an integrated change instruction via a single large-scale language model response and atomic application of the instruction), while Figure 8 mainly shows the message flow corresponding to Form B (multiple tool calls and responses, application for each call, and consistency verification after completion).

[0080] Step (1): This step involves receiving instructions from the user regarding changes to the application. The user inputs instructions using one of the following modalities: text input, voice input, or image input. In the case of text input, instructions in natural language are sent directly to the large-scale language model. In the case of voice input, the instructions are converted to text by speech recognition processing and then processed in the same way as text input. In the case of image input, images taken or attached by the user (such as screenshots of the application's current display screen, sketches of design drafts, or captures of reference UIs) are sent to the large-scale language model as visual information, and change commands are generated based on the image content. Input combining multiple modalities (e.g., simultaneous transmission of voice instructions and reference images) is also possible.

[0081] Step (2): A step of generating an abstract world state by extracting configuration information related to the instruction from the information contained in the integrated application state and the execution state of the runtime environment, thereby reducing the amount of information.

[0082] Step (3): A step of sending the instructions and the abstracted world state to a large language model. The contents of the abstracted world state to be sent may include a serialized summary format (Form II) when primarily using a batch format (Form A), and a tool-defined format (Form I) when primarily using a sequential format (Form B).

[0083] Step (4): A step in which the large-scale language model receives a change instruction. The change instruction may be an instruction that includes a change to a single component layer, or it may be an integrated change instruction that includes a change spanning at least two component layers. In batch form (form A), in response to the transmission in step (3), the large-scale language model returns an integrated change instruction as a single response that includes the changes spanning the multiple component layers in a single JSON structure. In sequential form (form B), the large-scale language model returns multiple individual tool calls based on the tool definition, and the AI ​​agent interface 140 interprets and processes each of these calls sequentially, thereby constructing an integrated change instruction that includes the changes spanning at least two component layers.

[0084] Step (5): A step to update the integrated application state while maintaining the execution state of the runtime environment by tracking the reference relationships included in the integrated application state and applying the changes while maintaining the consistency of said reference relationships. In the batch form (form A), the integrated update control unit 130 atomically applies the entire integrated change instruction received in step (4) in a batch. In the sequential form (form B), the integrated update control unit 130 sequentially applies the changes corresponding to each tool call in step (4), and after the completion of the series of tool calls, inter-layer consistency verification is performed to ensure the overall consistency of the integrated change instruction.

[0085] (Generating an abstract world state) In order for the AI ​​agent to properly generate changes, it is necessary to efficiently present the state of the runtime environment. In this embodiment, only the parts related to instructions are extracted from the integrated application state and the runtime environment execution state, and a state representation (abstracted world state) with reduced information volume is generated. The abstracted world state may be a state representation that holds only the information necessary for generating changes to the parts of the integrated application state and the runtime environment execution state related to instructions.

[0086] Figure 9 is a flowchart of the abstraction world state generation process. When the abstraction world state generation process starts, the world state generation unit 141 first acquires state information from the runtime environment 120 (step S11). The state information acquired here includes at least (1) the current integrated application state (GUI configuration information 111, driver configuration information 112, and logic configuration information 113), (2) the dynamic state of the runtime environment 120 (current value of each GUI element, driver connection state, state of the currently executing logic node), and (3) type definition information of the integrated application state. Next, the dependency graph construction unit 142 constructs the instance reference relationships between the GUI configuration information 111, driver configuration information 112, and logic configuration information 113 as a dependency graph G=(V,E) based on the acquired integrated application state (step S12). Note that E can be defined as a set of directed edges corresponding to instanceId references, next references, and data flow references. Next, the relevance determination unit 143 identifies a set of starting nodes on the dependency graph based on the user's instructions (step S13). Then, the relevance determination unit 143 extracts a set of related nodes consisting of nodes of configuration information related to the instructions from the set of starting nodes by performing a breadth-first search using a predetermined search radius (upper limit of the number of hops) (step S14). At this time, the search radius can be dynamically adjusted according to the scale of the integrated application state. Next, the world state generation unit 141 selects information corresponding to the extracted set of related nodes and integrates static configuration information and dynamic execution state information to form an intermediate representation that will be used to generate the abstracted world state (step S15). Subsequently, the world state generation unit 141 determines, based on the intermediate representation, whether the number of input tokens to the large-scale language model is less than or equal to a predetermined upper limit (step S16). If the determination result is negative (step S16; No), the abstraction transformation unit 144 reduces the amount of information by increasing the level of abstraction according to the type of configuration information (step S17), and then executes step S16 again. On the other hand, if the judgment result is positive (step S16; Yes), the world state generation unit 141 outputs an abstracted world state (step S18).Here, the abstracted world state may include only one of either the tool-defined format (Form I) or the serialized summary format (Form II), or it may include both. Finally, the world state generation unit 141 terminates the process of generating the abstracted world state.

[0087] Figure 10 is a conceptual diagram illustrating the concepts of constructing a dependency graph and extracting related nodes. As shown in Figure 10, the nodes of GUI configuration information, logic configuration information (AST), and driver configuration information are connected by directed edges that have a direction from the reference source to the reference destination, and a set of related nodes can be extracted from a starting point according to the user's instructions using a breadth-first search. In Figure 10, the extracted related nodes and the directed edges connecting them are shown in gray so that they can be visually distinguished from other nodes and edges.

[0088] The dependency graph construction unit 142 constructs the instance reference relationships between the GUI configuration information 111, driver configuration information 112, and logic configuration information 113 in the integrated application state as a dependency graph G=(V,E). Here, V is the set of all instances and all AST nodes in the integrated application state, and E is the set of directed edges by instanceId references, next references, and data flow references.

[0089] The relevance determination unit 143 identifies a set of starting nodes on the dependency graph based on the user's instructions. By setting a predetermined search radius (upper limit of the number of hops) and performing a breadth-first search from the set of starting nodes, it identifies a set of nodes containing configuration information related to the instructions. The search radius is dynamically adjusted according to the scale of the integrated application state. Therefore, in this embodiment, deterministic related part extraction based on reachability on the dependency graph is achieved and does not rely on heuristic search methods.

[0090] The abstraction transformation unit 144 transforms the configuration information corresponding to the extracted set of related nodes at a level of abstraction appropriate to the type of configuration information, so that the number of input tokens to the large-scale language model falls below a predetermined upper limit. Specifically, information is reduced in the following order: for GUI configuration information, the position and size information is simplified; for driver configuration information, connection parameters are omitted; and for logic configuration information, it is converted into a summarized representation of the AST node sequence.

[0091] By transforming from abstraction level L0 to L3, the amount of input information can be gradually compressed while retaining the parts necessary for change judgment. Here, abstraction level L0 is a standard representation of the integrated application state serialized, and abstraction levels L1 to L3 are representations obtained by gradually applying information reduction (e.g., exclusion of part type definitions, simplification of the serialized representation, and transformation of AST node sequences into a summarized representation) according to the type of configuration information to the standard representation. As a result, this embodiment achieves a qualitative technical improvement by combining related part extraction based on dependency graphs and abstraction transformation for each configuration type, thereby controlling the input to a large-scale language model within a practical range. Figure 11 shows a specific example of the change in the amount of input information before and after abstraction transformation using the serialized summary format (Form II). In the example shown in Figure 11, for a medium-sized application including 10 GUI configuration information instances, 3 driver configuration information instances, 75 logic configuration information AST nodes, 2 variables, and 4 lists, the number of tokens was 29,898 at abstraction level L0 and 13,303 at abstraction level L3, resulting in a reduction of approximately 55 percent. The numerical values ​​are measurement results from one embodiment and do not limit the present invention.

[0092] (Format and processing flow of integrated change instructions) Figure 12 is a flowchart of the atomic application process for an integrated change instruction. When the atomic application process for an integrated change instruction is started, first the transaction control unit 134 creates a snapshot of the integrated application state (step S21). Next, the integrated update control unit 130 determines the application order of the changes included in the integrated change instruction based on the dependency graph (step S22). Then, the consistency verification unit 132 verifies the referential integrity based on the integrated change instruction (step S23). If the referential integrity verification result is negative (step S23; No), the integrated update control unit 130 rejects the integrated change instruction (step S28), restores the integrated application state based on the snapshot (step S29), and then terminates the atomic application process. On the other hand, if the referential integrity verification result is positive (step S23; Yes), the safety verification unit 133 verifies safety, including at least one of infinite loops, runaways, and resource overloads (step S24). If the safety verification result is negative (step S24; No), the integrated update control unit 130 rejects the integrated change instruction (step S28), restores the integrated application state based on the snapshot (step S29), and then terminates the atomic application process. On the other hand, if the safety verification result is positive (step S24; Yes), the integrated update control unit 130 applies the changes to each component layer to the runtime environment 120 as a selective memory update according to the determined order (step S25). Subsequently, the integrated update control unit 130 determines whether the application of all changes was successful (step S26). If the determination result is negative (step S26; No), the integrated update control unit 130 rolls back the integrated application state based on the snapshot (step S29), and then terminates the atomic application process. On the other hand, if the determination result is positive (step S26; Yes), the reference integrity maintenance function 131 performs reference integrity maintenance, which includes at least one of updating, invalidating, and rejoining the remaining reference to an alternative instance (step S27), and the atomic apply process ends.

[0093] An integrated change instruction is a sub-concept of the aforementioned change instruction and is implemented in either the batch form (Form A) or the sequential form (Form B) described above. The specific processing flows for each form described below are concrete examples of steps (4) and (5) of the AI ​​agent collaboration pipeline and correspond to the sequence diagrams in Figure 7 (Form A) and Figure 8 (Form B).

[0094] (Form A) In the batch processing flow, the AI ​​agent interface 140 sends the abstracted world state and user instructions to the large language model, which then generates a JSON structure (integrated change instruction) containing changes across all configuration layers in a single response. The AI ​​agent interface 140 receives this response and passes it to the integrated update control unit 130 as an integrated change instruction. The integrated update control unit 130 performs referential integrity verification by the consistency verification unit 132, safety verification by the safety verification unit 133, determines the application order based on the dependency graph, and creates a snapshot by the transaction control unit 134, before atomically applying the changes that span multiple configuration layers. If any verification fails, the integrated change instruction as a whole is rejected, and the integrated application state is restored to the state before the change based on the snapshot. The process shown in Figure 12 is a detailed explanation of the application process in this batch form.

[0095] (Form B) In the sequential processing flow, the tool definitions (function calling schema) provided to the large-scale language model define operations on each component layer of the integrated application state as individual tools. Specifically, tools for GUI configuration information (create_parts, update_parts, delete_parts, list_parts), tools for driver configuration information (create_driver, update_driver, delete_driver, list_drivers), and tools for logic configuration information (create_ast_node, move_ast_node, change_ast_node, delete_ast_node) are defined. The large-scale language model autonomously combines and calls these tools to achieve changes that span multiple component layers. Here, the provided tool definitions can be dynamically updated to follow the components and user-added driver types present in the integrated application state.

[0096] The sequential tool execution loop is controlled based on the stop reason included in the response from the large language model. If the stop reason indicates a tool call request (tool_use), the AI ​​agent interface 140 extracts the tool call block included in the response, identifies and executes the corresponding tool implementation registered in the tool registry, sends the execution results of each tool to the large language model, and requests a response again. If the stop reason indicates a final response (end_turn), the tool execution loop is terminated, the final response is presented to the user, and a consistency check is performed on the entire integrated change instruction realized by the series of tool calls.

[0097] The tool registry is a component responsible for registering and managing available tool implementations. In one embodiment, the tool registry includes an adapter layer that absorbs differences in tool definition schema formats depending on the large-scale language model provider (Anthropic Claude, OpenAI GPT®, Google Gemini, etc.). This ensures extensibility, allowing switching of large-scale language model providers without changing the tool implementation itself. Tools registered in the tool registry can include not only system-embedded tools (tools for manipulating the state of integrated applications) but also user-defined external tools (queries to external services, file operations, etc.).

[0098] (Transaction control and event-driven architecture) When applying changes that span multiple configuration layers to a runtime environment that is currently executing, if the intermediate state of the change is observed, reference inconsistencies or execution errors may occur. Therefore, in this embodiment, it is necessary to ensure the atomicity and consistency of the change process. For this reason, the transaction control unit creates a snapshot of the integrated application state before applying the change instruction, and if an inconsistency or abnormality is detected during the change process, it restores the state based on the snapshot.

[0099] Figure 13 is a state transition diagram showing the state transitions of transaction control in non-destructive execution context updates. As shown in Figure 13, the runtime environment 120 waits for the arrival of an integrated change instruction in the normal execution state. When the integrated change instruction arrives, the transaction control unit 134 creates a snapshot of the integrated application state to enable rollback, and then sets a thread barrier to make each thread wait until a safe point. After reaching the safe point, the transaction control unit 134 generates a new object, switches the pointer by switching the mapping between the configuration information ID and the memory address in a batch, and then releases the barrier to return to the normal execution state. If the update is successful, the old object is not immediately destroyed but released with a delay, and if the update fails, it is restored based on the snapshot. At each stage of this series of transitions, the runtime environment 120 can be observed from the outside as if execution is continuing.

[0100] In this embodiment, all operations on the runtime environment 120 are processed based on an event-driven architecture in which serializable event data objects are issued as boundary representations. Specifically, manual editing operations by the user via the integrated editor 150, autonomous modification operations by the AI ​​agent via the AI ​​agent interface 140, and remote operations transmitted from an external system via the network are all represented in the same event data object format and processed uniformly on the same event processing infrastructure of the integrated update control unit 130.

[0101] Figure 14 is a timing diagram showing the operation of event processing between threads. As shown in Figure 14, coordination is performed between the main thread, runtime thread, and AI agent-dedicated thread by an event distribution and synchronization mechanism based on the publish type described later, thereby maintaining consistency between configuration information updates and runtime execution. In this embodiment, the main thread is responsible for drawing and updating the user interface and receiving user input, the runtime thread is responsible for sequentially executing the application by the runtime environment 120 and updating the integrated application state by the integrated update control unit 130 on the same thread, and the AI ​​agent-dedicated thread is responsible for communication with the large-scale language model and tool execution, etc. Each thread maintains an independent event queue, and state sharing between threads is performed exclusively via event data objects to which a publish type has been assigned. As a synchronization mechanism between these threads, an event queue using condition variables and mutexes, a thread barrier, or a combination thereof can be used, and in any case, the application execution process and the integrated application state update process in the runtime thread are processed sequentially on the same thread, thereby suppressing the observation of intermediate states such as the replacement of referenced objects by other threads.

[0102] This unification of the operation path yields the following technical benefits: Firstly, the same consistency and security verifications are applied regardless of the origin of the operation (user, AI agent, remote system). Secondly, remote operations over the network are represented in the same event data object format as local operations, thus separating the communication layer from the control logic layer. Thirdly, since event data objects are serializable, recording operation history, replaying operations, and undoing operations are uniformly implemented as operations on the event log.

[0103] Each event data object is assigned a transmission type (publish type). Events to be processed by the main thread, runtime thread, and AI agent-dedicated thread are distributed to the appropriate thread's event queue based on the publish type. In one embodiment, the publish type includes types corresponding to the main thread, runtime processing, drawing processing, and AI agent processing, and is assigned according to the source and purpose of the event. A synchronous processing request from one thread to another is realized in which the requesting thread issues a request event with a request identifier, waits for a response using a synchronization mechanism (e.g., an event queue using condition variables and mutexes, a thread barrier, or a combination thereof), and the requesting thread obtains the processing result when the responding thread issues a corresponding response event after processing is complete. In one embodiment, the AI ​​agent-dedicated thread waits to process the result of the next tool call until it receives a response event indicating that the integrated application state update by the runtime thread is complete.

[0104] (Integrated Update Control Unit) The integrated update control unit 130 performs a series of processes based on the change instruction, including dependency analysis, consistency maintenance, verification, and update application. Specifically, the integrated update control unit 130 sequentially performs the following: determining the application order based on dependency analysis, maintaining referential integrity, verifying consistency and safety, and applying the update through transaction control.

[0105] First, the integrated update control unit 130 analyzes the reference relationships between instances of GUI configuration information, driver configuration information, and logic configuration information included in the integrated application state and generates a dependency graph representing these dependencies. Next, the integrated update control unit 130 searches the dependency graph starting from the node corresponding to the instance modified by the change instruction and identifies the configuration information corresponding to other nodes that depend on that node. Based on the dependency graph, the integrated update control unit 130 determines the application order of the multiple updates included in the change instruction.

[0106] For example, in an integrated change instruction that includes adding a driver instance and adding an AST node that references that driver, the direction of the instanceId reference edge in the dependency graph leads to the application order in which the addition of the driver instance precedes the addition of the AST node.

[0107] Subsequently, the reference consistency maintenance function 131 of the integrated update control unit 130 controls the update process to maintain the consistency of reference relationships when applying changes according to the determined application order. The reference consistency maintenance function 131 automatically updates, invalidates, or reconnects references to GUI configuration information or driver configuration information instances in the logic configuration information in response to the addition, modification, or deletion of such instances by the change command.

[0108] When an AI agent generates changes across multiple configuration layers simultaneously, the following referential integrity issues arise. Firstly, as a referential integrity issue associated with multi-layer simultaneous changes, the dependency of the operation order must be automatically controlled in change instructions that include adding drivers corresponding to the addition of GUI elements and adding logic to control those drivers. Secondly, as a residual reference issue associated with the deletion of components, when a GUI element or driver instance is deleted, the reference (identifier) ​​to the deleted element in the logic configuration information must be identified and handled appropriately.

[0109] Figure 15 is a flowchart showing the process of updating or invalidating a reference by the reference consistency maintenance function 131. First, the reference consistency maintenance function 131 enumerates all AST nodes that reference the instanceId of the deleted or modified instance by traversing the reverse edge of the dependency graph G (direction from the referenced node to the referencing node) (step S31). Next, for each enumerated AST node, the reference consistency maintenance function 131 sequentially determines whether a replacement instance with the same driver type or UI element type as the deleted instance can be uniquely identified within the integrated application state (step S32). If the determination result is positive (step S32; Yes), the reference consistency maintenance function 131 updates the instanceId reference of the AST node to the instanceId of the replacement instance (step S33). On the other hand, if the determination result is negative (step S32; No), the reference consistency maintenance function 131 invalidates the reference by setting the instanceId reference of the AST node to null (step S34). Next, the consistency verification unit 132 performs a final referential integrity check after the completion of the reference update or invalidation of the enumerated AST nodes (step S35). If the result of the final check is negative (step S35; No), the transaction control unit 134 performs a rollback (step S36) and terminates the process. If the result of the final check is positive (step S35; Yes), the process terminates immediately.

[0110] Furthermore, the integrated update control unit 130 performs consistency verification and safety verification before or during the application of the changes.

[0111] The consistency verification unit 132 verifies the referential integrity between the GUI configuration information, driver configuration information, and logic configuration information before applying the change instruction. The consistency verification unit 132 rejects the application of the change instruction if it detects at least one of the following: a reference to a non-existent instance, type mismatch, or unconnected input / output.

[0112] The safety verification unit 133 verifies, before applying a change instruction, that the change to the logic configuration information will not cause an infinite loop, runaway control signals via the driver configuration information, or exceed a predetermined resource consumption limit. In particular, in environments where hardware devices are controlled via driver configuration information, incorrect changes to the AI ​​agent can cause physical damage, making safety verification before applying changes essential. In cloud deployment configurations, additional verification is performed to ensure limits on API call frequency, prevention of authentication information leakage, and compliance with CORS (Cross-Origin Resource Sharing) policies.

[0113] The verification process of the safety verification unit 133 consists of the following three stages.

[0114] (1) Structural cycle detection: Depth-first search (DFS) is used to detect back edges in the directed graph formed by the AST node group of logic configuration information. This directed graph consists of the next reference of each AST node, as well as the input references held by the control flow AST node. The presence of a back edge indicates that the connection structure of the AST nodes contains unintended circular references. Note that intentional repetitive operations of control flow AST nodes themselves, such as repeating AST nodes, are execution controls managed by the script execution unit 123 of the runtime environment 120 and are distinct from cycles in the connection structure of the AST nodes.

[0115] (2) Driver transmission frequency limit: The AST node column of the logic configuration information verifies that the number of transmission operations to the same driver instance does not exceed a predetermined limit within a predetermined time window. If control signals are sent to the hardware device control driver consecutively at intervals less than a predetermined minimum, it is detected as a control signal malfunction. For software service connection drivers, it is verified that the frequency of requests to external APIs does not exceed the rate limit of the service provider.

[0116] (3) Execution step limit: This verifies that the number of AST nodes on the execution path starting from a single event-starting AST node does not exceed a predetermined maximum number of steps. Since the termination of a conditional loop is generally undecidable (termination problem), complete infinite loop detection cannot be theoretically guaranteed. This verification functions as a practical protection mechanism against this theoretical limit, with the safety verification unit 133 monitoring the number of steps during execution and forcibly halting execution using a snapshot of the transaction control unit 134 if the limit is exceeded.

[0117] The safety verification unit 133 may include tool permission control for tool operations performed by the AI ​​agent. In one embodiment, each tool is assigned one of three permission levels: always permitted, require confirmation each time, or always denied. Always permitted applies to read operations that reference the state of the integrated application (e.g., obtaining a parts list, obtaining a driver list). Required confirmation applies to write and delete operations that modify the state of the integrated application (e.g., creating parts, updating drivers, deleting AST nodes), and requires user approval before execution. Always denied applies to operations prohibited by the system administrator or user. This tool permission control allows the user to explicitly control the scope of autonomous operations by the AI ​​agent, and in particular, prevents unintended changes when the AI ​​agent modifies an application running in a production environment.

[0118] Furthermore, in this embodiment, since the integrated application state has reference relationships between GUI configuration information, driver configuration information, and logic configuration information, changes to one component may affect other configuration layers. Therefore, change processing needs to be managed not as a single component unit, but as an integrated unit that takes dependencies into account.

[0119] Finally, the integrated update control unit 130 applies changes spanning multiple component layers as a single unit, ensuring atomicity of the changes through transaction control, and updating the integrated application state while maintaining the execution state of the runtime environment.

[0120] The transaction control unit 134 creates a snapshot of the integrated application state before applying a change instruction, and restores the integrated application state based on the snapshot if an anomaly is detected during the update process. This prevents the persistence of an inconsistent state where changes have been partially applied.

[0121] Furthermore, the integrated update control unit 130 identifies the memory area of ​​the runtime environment corresponding to the configuration information specified by the change command and selectively updates only that memory area. As a result, the state of the runtime environment corresponding to the configuration information that is not to be changed (running processes, established communication connections, execution context, etc.) is preserved without being discarded.

[0122] Figure 16 shows the structure of the mapping table between configuration information IDs and memory addresses. As shown in Figure 16, selective memory updates are realized by the following three mechanisms.

[0123] (1) Configuration Information ID-Memory Address Mapping Table: The runtime environment 120 maintains a mapping table that holds the correspondence between the instanceId of each instance in the integrated application state and the memory address range allocated to that instance on the virtual machine heap. When an integrated change instruction is received, the transaction control unit 134 refers to the mapping table using the instanceId included in the instruction as the key and dynamically identifies the memory area to be updated.

[0124] (2) Replacing objects on the virtual machine heap: For the instance to be updated, an object with new attribute values ​​is created on the virtual machine heap, and the pointer in the mapping table is switched to the new object. The old object is released when it is no longer referenced by the running thread. This method replaces only the memory area of ​​the instance to be updated, and the memory areas of other instances are not affected.

[0125] (3) Synchronization control by thread barrier: In order to atomically apply updates that span multiple configuration layers, the transaction control unit 134 sets a thread barrier before applying the update and, after confirming that all execution threads in the runtime environment 120 have reached a safe point, performs a batch of pointer switching for the mapping table. This prevents the intermediate state of the update process from being observed by other threads and ensures that only the state before or after the change can be observed.

[0126] (Iterative Improvement Loop) Figure 17 is a flowchart illustrating the operation of the iterative improvement loop and convergence control. First, the AI ​​agent interface 140 generates a change command according to the AI ​​agent collaboration pipeline shown in Figure 7 or Figure 8 (step S41). Next, the integrated update control unit 130 applies the change command to the integrated application state (step S42). Subsequently, the runtime environment 120 executes the application based on the integrated application state after application (step S43). Next, the world state generation unit 141 generates an updated abstract world state including the execution result (step S44). Subsequently, the AI ​​agent interface 140 determines whether the convergence condition is met based on whether at least one of the following conditions is met: the amount of change in the abstract world state is below a predetermined threshold, the number of iterations is greater than or equal to a predetermined maximum number of iterations, or an anomaly is detected in the safety verification (step S45). If the determination result is positive (step S45; Yes), the iterative improvement loop terminates. On the other hand, if the judgment result is negative (step S45; No), the AI ​​agent interface 140 sends the updated abstracted world state to the large language model (step S46). The iterative improvement loop then returns to step S41.

[0127] The AI ​​agent interface 140 can, after applying a change instruction, execute the runtime environment based on the updated integrated application state and generate an updated abstraction world state including the execution results. Furthermore, after sending the updated abstraction world state to the large language model, it can iteratively perform application improvement processing by receiving new change instructions according to the AI ​​agent collaboration pipeline shown in Figure 7 or Figure 8.

[0128] The iterative improvement process terminates when at least one of the following conditions is met: the amount of change in the abstracted world state falls below a predetermined threshold, a predetermined maximum number of iterations is reached, or an anomaly is detected during safety verification.

[0129] The aforementioned "amount of change" is an indicator that quantifies the change in the abstracted world state before and after the update. For example, it can be measured as the number of AST nodes that have been modified before and after the application of the change instruction, or as the number of difference tokens in the text representation of the abstracted world state generated by the serialized summary format. Depending on the scale of the runtime environment 120 and the required convergence accuracy, these indicators may be used individually or in combination.

[0130] Figure 18 is a block diagram showing an example configuration of an application execution system 100 on a single computing device 160. As shown in Figure 18, in a computing device 160 equipped with a touch-sensitive display, a processor, and memory, all components of the application execution system 100 operate within a single application. In this embodiment, the integrated editor 150, runtime environment 120, and AI agent interface 140 operate within a single application on the same computing device 160, allowing the user to complete everything from giving instructions in natural language to modifying the application on that device 160.

[0131] (Tool execution loop in sequential form) In sequential mode (mode B), the AI ​​agent interface 140 controls the tool execution loop based on the response termination reason included in the response from the large-scale language model. If the response termination reason indicates a tool call request, the AI ​​agent interface 140 extracts the tool call block included in the response, executes each tool sequentially, sends the execution results to the large-scale language model, and requests a response again. If the response termination reason indicates a final response, the tool execution loop is terminated and the final response is presented to the user.

[0132] The tool execution loop has a maximum iteration limit. If the large-scale language model does not terminate the tool execution loop even after reaching the limit, the AI ​​agent interface 140 forcibly terminates the loop and performs a consistency check on the changes made by the tool calls executed up to that point. If an inconsistency is detected during the consistency check, the transaction control unit 134 performs a recovery process based on the snapshot.

[0133] In sequential mode, tool definitions provided to large-scale language models can also be selected from tools registered in a tool registry and provided as a subset of tools relevant to the user's instructions. This reduces the number of tool definitions processed by the large-scale language model, suppressing unnecessary tool calls and reducing token consumption.

[0134] In one embodiment, tool execution of the AI ​​agent interface 140 is performed on a dedicated thread for the AI ​​agent, and the application operation to the integrated application state is performed by the integrated update control unit 130 on the main thread. Each event is assigned a transmission type (publish type), and events for the dedicated thread for the AI ​​agent and events for the main thread are distributed to the appropriate thread based on the publish type. With this configuration, each tool call in sequential mode is applied only to the target configuration information of the integrated application state without interrupting the running state of the runtime environment (established BLE connection, ongoing HTTP communication, running logic, etc.).

[0135] In another embodiment, an adaptive mode is possible that dynamically switches between batch mode and sequential mode. Specifically, based on the scale and complexity of the changes estimated from the user's instructions, the batch mode is automatically selected if the changes are simple and limited, and the sequential mode is selected if the changes are complex or require multi-stage adjustments. This selection is performed in parallel with the abstraction world state generation process, and the result of the mode selection is reflected in the initial request to the large language model.

[0136] (Integrated editor and multi-platform deployment) The integrated editor 150 provides an interface that enables editing of GUI configuration information 111, driver configuration information 112, and logic configuration information 113 on a touch-sensitive display. The integrated editor 150 includes the following functions:

[0137] (1) Scene Graph Editing Function: A function to visually edit the scene graph, which defines the hierarchical arrangement of user interface elements. The scene graph includes a layout container, which defines automatic arrangement rules including the arrangement direction, spacing, and margins of its child elements.

[0138] (2) Dynamic interface generation function based on part type definitions: A function that dynamically generates an editing interface that reflects the name, data type, default value, range constraints, candidate options, and dependencies between attributes of each attribute, based on the self-descriptive attribute information of the part type definitions included in the integrated application state.

[0139] Figure 19 is a block diagram showing a configuration in which the same integrated application state can be deployed to multiple platforms. As shown in Figure 19, the same definition can be deployed to smartphones / tablets, web browsers, cloud servers, and single-board computers, and the runtime environment 120 can select the rendering backend.

[0140] The same integrated application state can be executed by the runtime environment 120 on the following platforms without requiring any modification of the integrated application state.

[0141] (1) Smartphones and tablets: Deployed as native applications for iOS (Swift / Objective-C) and Android (Kotlin / Java). GUI rendering engine 121 selects native rendering using Metal (iOS) or OpenGLES (Android) for controller part types in category A, and HTML rendering via the WebView component for web / document element types in category B.

[0142] (2) Web browser: The runtime environment 120 is compiled into WebAssembly or JavaScript and runs on the browser. Elements of category B of the GUI configuration information are rendered directly as HTML / CSS, while elements of category A are rendered on canvas or WebGL. The AI ​​agent interface 140 runs on a WebWorker within the browser, and API communication with the large language model is performed without interfering with the main thread.

[0143] (3) Cloud Server: The runtime environment 120 operates within a container (such as Docker) and outputs GUI configuration information 111 as server-side rendering (SSR) or API response. In the case of SSR, the runtime environment 120 outputs the elements of category B of the GUI configuration information as HTML strings, and can be operated as an interactive web application through hydration processing on the client side. Category II drivers in the driver configuration information are responsible for communication with backend services, and category III drivers provide server-side utility functions.

[0144] (4) Single-board computers and microcontrollers: Native implementations in C / C++ are provided for Raspberry Pi (Linux®) and ESP32 (FreeRTOS). In resource-constrained environments, a headless mode is also available that minimizes the overhead of the GUI rendering engine 121 by omitting the rendering of GUI configuration information and executing only driver configuration information and logic configuration information.

[0145] The fact that the same integrated application state can be executed by the runtime environment 120 on each of the above platforms (1) to (4) is due to the fact that the integrated application state is defined as an abstract data representation that does not depend on a specific drawing backend or OSAPI. The runtime environment 120 detects the deployment target platform at startup and dynamically selects the implementation of the GUI drawing engine 121 and the communication management unit 122 based on the available drawing backend and OSAPI. Since this implementation selection does not require a change in the integrated application state, developers can support multiple platforms by defining a single integrated application state. [Examples]

[0146] The following describes examples that specifically illustrate the effects of the present invention. [Examples]

[0147] (Adding an emergency stop button to drone control applications) This embodiment describes the operation of the AI ​​agent collaboration pipeline when a user instructs a drone control application (a BLE control application equipped with a joystick and forward buttons) to "add an emergency stop button."

[0148] The AI ​​agent interface 140 constructs a dependency graph from the execution state of the runtime environment 120, performs a breadth-first search starting from the node related to "emergency stop" (driver instance: ble_drone_01), generates an abstracted world state, and sends it to the large language model. The large language model generates an integrated change instruction that includes adding an emergency stop button (instanceId: button_emergency) to the GUI configuration information and adding a script to handle the button press event to the logic configuration information.

[0149] The integrated update control unit 130 performs verification by the consistency verification unit 132 and verification by the safety verification unit 133, and then atomically applies the integrated change command in the application order based on the dependency graph (addition of GUI elements → addition of logic configuration information). The main thread of the runtime environment 120 immediately reflects the display of the emergency stop button and its event processing logic while maintaining the BLE connection. [Examples]

[0150] (Integration changes to the web dashboard application) Figure 20 is a conceptual diagram illustrating the concept of integrated modifications to the web dashboard application in this embodiment. As shown in Figure 20, changes spanning three layers, such as the addition of a graph container, an HTTP client driver, and periodic retrieval logic, can be applied while maintaining the existing API connectivity.

[0151] This embodiment describes the operation of a web dashboard application that displays sensor data when a user instructs it to "add a real-time graph and retrieve data from the API every 5 seconds." The large-scale language model generates an integrated change command that includes the addition of a web / document element type graph container element (GUI configuration information), the addition of an HTTP client driver instance (driver configuration information), and the addition of periodic retrieval and graph update logic (logic configuration information). The integrated update control unit 130 atomically applies the changes across the three configuration layers while maintaining the existing API connection.

[0152] (Modified examples of the embodiment) While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. For example, responses from a large-scale language model may be received sequentially using a streaming protocol (such as HTTP / 2 Server-Sent Events or WebSocket), and integrated change instructions may be applied in stages. In this case, by starting to apply the previously received changes to the integrated application state before the large-scale language model's response is complete, the delay in change reflection experienced by the user can be reduced. A snapshot may be created by the transaction control unit 134 before each partial application, and if a subsequent partial application is rejected by consistency verification, a rollback to the previous snapshot may be performed.

[0153] Furthermore, in generating the abstracted world state, the starting node identification process of the relevance determination unit 143 may be combined with relevance weighting based on past change history or estimation of the starting node based on semantic analysis of user instructions. Specifically, by setting a high relevance weight for a set of nodes where changes have been concentrated in the past and adjusting the priority of the search range, it is possible to learnfully improve the process so that a more appropriate abstracted world state is generated for the same user instruction.

[0154] Alternatively, instead of API communication with a large-scale language model, a locally executed small language model (SLM) may be used. Using a local SLM eliminates the need for network communication, allowing the system to be applied to embedded devices requiring network connectivity or offline operation. Furthermore, since the application's integrated state is not transmitted to an external server, it offers security advantages when applied to applications containing confidential information. The local SLM may use its own tool calling format (e.g., XML tag format).<tool_call> If using elements, the tool definition schema may be converted using the tool registry's compatibility layer.

[0155] Furthermore, the AI ​​agent may autonomously monitor the application's execution status without explicit user instructions and automatically propose and apply changes for performance optimization or error recovery. In this case, the AI ​​agent interface 140 can periodically generate an abstracted world state from the execution status of the runtime environment 120 and autonomously determine whether changes are necessary by comparing it with pre-configured performance indicators or error state definitions. When applying changes, the safety verification unit 133 may be configured by default to require user approval each time an autonomous change is made by the AI ​​agent, by setting the tool permission control to require confirmation each time.

[0156] Furthermore, the GUI configuration information 111 may include a third category corresponding to primitives of native mobile UI elements (SwiftUI for iOS, Jetpack Compose for Android), in addition to controller part types of category A and web / document element types of category B. Since the instanceId reference mechanism for the integrated application state does not depend on the category of the UI element, the addition of a new category can be achieved simply by adding a part type definition and implementing the corresponding rendering backend. This allows the expressive power of the GUI configuration information of this system to be expanded in stages in accordance with the evolution of the platform and UI framework.

[0157] Furthermore, a build pipeline may be provided that performs static conversion from the integrated application state to HTML / CSS / JavaScript, generating a static site that can be deployed to a web server without going through runtime environment 120. In this configuration, a workflow can be achieved where real-time execution by runtime environment 120 and iterative improvement by an AI agent are utilized during the development and testing phases, and optimized HTML / CSS / JavaScript is output for production deployment. In this case, runtime environment 120 operates only in the development environment, and static files are delivered from a CDN (Content Delivery Network) in the production environment, thus eliminating the overhead of the runtime environment from the production environment.

[0158] Furthermore, as user instruction input, images captured by a camera may be sent to a multimodal large-scale language model to recognize the design intent of user interface elements contained in the image and generate integrated modification commands. For example, UI elements such as buttons, sliders, and text display areas can be detected from a photograph of a handwritten UI design on a whiteboard, their spatial arrangement can be analyzed to generate a scene graph, and then combined with the user's supplementary text instructions to generate integrated modification commands spanning three layers: GUI configuration information, driver configuration information, and logic configuration information. This multimodal input can be implemented as a form of image input modality in step (1) of the AI ​​agent collaboration pipeline described above, where user instructions are received.

[0159] Furthermore, in a configuration that implements integrated modification instructions in a sequential form (form B), a compatibility layer may be provided to absorb the differences in tool invocation protocols provided by providers of large-scale language models. Specifically, it may include a serializer that internally holds a unified tool definition (tool name, description, parameter schema) and converts it to a schema format corresponding to the provider of the large-scale language model (e.g., input_schema format for Anthropic API, function.parameters format for OpenAI API, functionDeclarations format for Google Gemini API), and a parser that normalizes the tool invocation response format of each provider into a unified tool invocation expression. The compatibility layer in this modified example can be implemented as the adapter layer of the tool registry described above.

[0160] The embodiments described above are illustrative, and any configuration that is substantially identical to the technical idea described in the claims of the present invention and achieves similar effects is included within the technical scope of the present invention. In other words, modifications can be made as appropriate within the scope of the technical idea expressed in the present invention, and such modified or improved forms are also included within the technical scope of the present invention. [Explanation of Symbols]

[0161] 100 Application Execution Systems 110 Application-Defined Models 111 GUI configuration information 112 Driver Configuration Information 113 Logic Configuration Information 120 Runtime Environments 121 GUI rendering engines 122 Communications Management Department 123 Script Execution Unit 130 Integrated Update Control Unit 131 Referential integrity maintenance function 132 Consistency Verification Department 133 Safety Verification Department 134 Transaction Control Unit 140 AI Agent Interfaces 141 World State Generation Unit 142 Dependency Graph Construction Section 143 Relevance Determination Unit 144 Abstraction Transformation Unit 150 Integrated Editors 160 computing devices

Claims

1. An application execution system that runs on a computing device, An application definition model that maintains GUI configuration information defining user interface elements, driver configuration information abstracting access to functions or resources used by logic configuration information, and logic configuration information defining the behavior of the application by referencing at least one of the GUI configuration information and the driver configuration information, as an integrated application state having mutual reference relationships. A runtime environment that interprets and executes the aforementioned integrated application state without going through the compilation or build process, An integrated update control unit receives a change command from an AI agent, which includes changes to at least two of the GUI configuration information, driver configuration information, and logic configuration information, and updates the integrated application state by applying the change command while maintaining the consistency of the reference relationships, while tracking the reference relationships between the GUI configuration information, driver configuration information, and logic configuration information included in the integrated application state. An AI agent interface that presents the abstracted world state generated from the integrated application state to the AI ​​agent, receives the change command generated by the AI ​​agent and transmits it to the integrated update control unit, Equipped with, The integrated update control unit updates the integrated application state while maintaining the execution state of the runtime environment. Application execution system.

2. The application execution system according to claim 1, wherein the integrated update control unit includes a reference integrity maintenance function that automatically updates, invalidates, or reconnects to an alternative instance of the GUI configuration information or driver configuration information in response to the addition, modification, or deletion of an instance of the GUI configuration information or driver configuration information by the modification command.

3. The application execution system according to claim 1, wherein the integrated update control unit includes a consistency verification unit that verifies the referential integrity between the GUI configuration information, the driver configuration information, and the logic configuration information before applying the change command.

4. The application execution system according to claim 3, wherein the consistency verification unit refuses to apply the modification instruction if it detects at least one of the following: a reference to a non-existent instance, a type mismatch, and an unconnected input / output.

5. The application execution system according to claim 1, further comprising a safety verification unit that verifies, before applying the change command, that the change in the logic configuration information does not cause an infinite loop, a runaway control signal via the driver configuration information, or an exceedance of a predetermined resource consumption limit.

6. The application execution system according to claim 1, wherein the integrated update control unit generates a dependency graph representing the reference relationships between instances of the GUI configuration information, the driver configuration information, and the logic configuration information included in the integrated application state.

7. The application execution system according to claim 6, wherein the integrated update control unit searches the dependency graph starting from the node corresponding to the instance to be modified by the change command and identifies configuration information corresponding to other nodes that depend on that node.

8. The application execution system according to claim 6, wherein the integrated update control unit determines the application order of a plurality of updates included in the change instruction based on the dependency graph.

9. The application execution system according to claim 6, wherein the AI ​​agent interface generates the abstracted world state by a process that deterministically extracts a set of nodes reachable from the starting node on the dependency graph based on the search radius.

10. The application execution system according to claim 1, wherein the integrated update control unit identifies a memory area of ​​the runtime environment corresponding to the configuration information specified by the change command and selectively updates only that memory area.

11. The application execution system according to claim 1, wherein the integrated update control unit includes a transaction control unit that creates a snapshot of the integrated application state before applying the change command, and restores the integrated application state based on the snapshot if an abnormality is detected during the update process.

12. The application execution system according to claim 1, wherein the logic configuration information is represented as an execution description structure consisting of interconnected nodes, and each node refers to an instance of a user interface element in the GUI configuration information or a driver instance in the driver configuration information by an identifier.

13. The application execution system according to claim 1, wherein the integrated application state is maintained as a serialized representation having the same structure as the execution representation of the runtime environment.

14. The application execution system according to claim 1, wherein the AI ​​agent interface dynamically updates the tool definition corresponding to a component in response to the addition, modification, or deletion of a component included in the integrated application state.

15. The application execution system according to claim 1, wherein operations on the runtime environment, whether manual editing operations by a user, modification commands by the AI ​​agent, or remote operations received from an external system, are all represented in a common serializable event data object format and received in the same processing path of the integrated update control unit.

16. The application execution system according to claim 1, wherein the change commands are issued sequentially from the AI ​​agent as a series of tool calls, and the integrated update control unit performs consistency verification to verify the integrity of the reference relationships after the completion of the series of tool calls.

17. A method for modifying an application in a runtime environment during execution, in cooperation with an AI agent, in which an integrated application state is included, which includes GUI configuration information that defines user interface elements, driver configuration information that abstracts access to functions or resources used by logic configuration information, and logic configuration information that defines the behavior of the application by referring to at least one of the GUI configuration information and the driver configuration information. Steps include receiving instructions from users regarding changes to the application, The steps include: generating an abstract world state by extracting configuration information related to the instruction from the information contained in the integrated application state and the runtime environment execution state, thereby reducing the amount of information; The steps include sending the aforementioned instructions and the abstracted world state to a large-scale language model, The steps include receiving a change instruction from the large-scale language model, which includes a change to at least two of the GUI configuration information, the driver configuration information, and the logic configuration information, The steps include updating the integrated application state while maintaining the execution state of the runtime environment by applying the change instruction while tracking the reference relationships between the GUI configuration information, the driver configuration information, and the logic configuration information included in the integrated application state and maintaining the consistency of said reference relationships, A method that includes this.

18. The method according to claim 17, wherein the step of generating the abstract world state includes a process of constructing a dependency graph of reference relationships between configuration information included in the integrated application state.

19. The method according to claim 18, wherein the step of generating the abstract world state includes a process of identifying a starting node on the dependency graph based on the user's instructions and extracting configuration information corresponding to nodes that exist within a predetermined search range from the starting node.

20. The method according to claim 18, wherein the step of generating the abstract world state includes a process of deterministically extracting a set of nodes reachable from a starting node on the dependency graph based on a predetermined search radius.

21. The method according to claim 18, wherein the step of generating the abstracted world state includes structuring the components identified by the dependency graph as tool definitions in the tool invocation protocol of the large language model.

22. The method according to claim 21, wherein the step of receiving the change instruction includes a process of verifying the conformity of the parameters of the tool call received from the large language model to the parameter schema of the tool definition.

23. The method according to claim 17, wherein the step of applying the change instruction includes a process of atomically applying a plurality of updates spanning two or more different configuration information from the GUI configuration information, the driver configuration information, and the logic configuration information as a single instruction.

24. The method according to claim 18, wherein the step of applying the change instruction includes a process of determining the order in which to apply a plurality of updates included in the change instruction based on the dependency graph.

25. The method according to claim 17, wherein the step of applying the change instruction includes identifying a memory region of the runtime environment corresponding to the configuration information to be changed by the change instruction, and selectively updating only that memory region.

26. The method according to claim 17, wherein the step of applying the change instruction further includes generating a snapshot of the integrated application state before updating the integrated application state, and restoring the integrated application state based on the snapshot if an anomaly is detected during the update process.

27. The method according to claim 17, further comprising the step of applying the change instruction, executing the runtime environment based on the updated integrated application state, and generating an updated abstraction world state including the result of said execution.

28. The method according to claim 27, further comprising the step of iteratively performing application improvement processing by sending the updated abstraction world state to the large language model and receiving new change instructions.

29. The method according to claim 28, wherein the iterative improvement process is terminated when at least one of the following conditions is met: the amount of change in the abstracted world state falls below a predetermined threshold, a predetermined maximum number of iterations is reached, or an abnormality is detected in the safety verification.

30. The method according to claim 17, wherein the step of generating the abstracted world state is performed as a summarization process for a serialized representation having the same structure as the execution representation of the runtime environment.

31. The method according to claim 17, wherein the step of applying the change command includes the step of converting user operations, AI agent operations and remote operations into a common event data object format and processing them.

32. A computing device comprising a touch-sensitive display, a processor, and memory, When the program stored in the memory is executed by the processor, the computing device, A process that provides a runtime environment for interpreting and executing an integrated application state, which includes GUI configuration information defining user interface elements, driver configuration information abstracting access to functions or resources used by logic configuration information, and logic configuration information defining the behavior of the application by referencing at least one of the GUI configuration information and the driver configuration information, without going through a compilation or build process. A process that provides an integrated editor capable of editing the GUI configuration information, the driver configuration information, and the logic configuration information, The AI ​​agent receives a change command that includes changes to at least two of the GUI configuration information, driver configuration information, and logic configuration information, and updates the integrated application state while tracking the reference relationships between the GUI configuration information, driver configuration information, and logic configuration information included in the integrated application state and maintaining the consistency of said reference relationships. Make it run, A computing device in which the integrated editor and the runtime environment operate within a single application on the computing device.

33. The aforementioned integrated editor includes a scene graph editing function that defines the hierarchical arrangement of user interface elements, The computing device according to claim 32, wherein the scene graph includes a layout container, and the layout container defines automatic placement rules including the placement direction, spacing, and margins of child elements.

34. The computing device according to claim 32, wherein the integrated application state further includes part type definitions that define attribute schemas for each type of user interface element, the part type definitions having self-descriptive attribute information including the name, data type, default value, range constraints, choice candidates and dependencies between attributes for each attribute, and the integrated editor dynamically generates an editing interface based on the part type definitions.

35. The computing device further comprises a function to construct a dependency graph of reference relationships between configuration information included in the integrated application state, extract configuration information corresponding to nodes related to the processing of the AI ​​agent to generate an abstracted world state with reduced information volume and transmit it to the AI ​​agent, wherein the abstracted world state includes at least one of the following: a form in which the extracted components are structured as tool definitions in a tool invocation protocol of a large-scale language model, and an information-reduced form of the serialized representation of the integrated application state.

36. The computing device according to claim 32, wherein, when applying the change instruction, it performs automatic updating of references in the logic configuration information in accordance with the change in the GUI configuration information or the driver configuration information, and verifies that the change in the logic configuration information does not cause an infinite loop or a runaway control signal.

37. The computing device according to claim 35, wherein the abstracted world state includes a tool-defined format that is dynamically updated in response to the addition, modification, or deletion of components included in the integrated application state.

38. The computing device according to claim 32, wherein the computing device processes manual editing operations by the user, change commands by the AI ​​agent, and remote operations received from an external system in a common serializable event data object format.

39. Computers, GUI configuration information that defines user interface elements, Driver configuration information that abstracts access to functions or resources used by logic configuration information, An application definition model that maintains logic configuration information that defines the behavior of an application by referencing at least one of the GUI configuration information and an instance of the driver configuration information, as an integrated application state having a mutually referential relationship; A runtime environment that interprets and executes the aforementioned integrated application state without going through the compilation or build process, An integrated update control unit receives a change command from an AI agent, which includes changes to at least two of the GUI configuration information, driver configuration information, and logic configuration information, and updates the integrated application state while tracking the reference relationships between the GUI configuration information, driver configuration information, and logic configuration information included in the integrated application state and maintaining the consistency of said reference relationships. An AI agent interface that presents the abstracted world state generated from the integrated application state to the AI ​​agent, receives the change command generated by the AI ​​agent and transmits it to the integrated update control unit, A program that enables the application execution system to function as such.

Images