Intention-driven mixed reality content automatic generation method, system and storage medium

Abstract

The application relates to the technical field of mixed reality, and discloses an intention-driven mixed reality content automatic generation method and system and a storage medium, which comprise the following steps: analyzing structured intention data to generate a machine-generated parameter set; calling a multi-modal large model matrix to process the machine-generated parameter set to generate a multi-modal asset set; reducing the dimension of an original three-dimensional entity file, writing the optimized three-dimensional entity file and a spatial audio file into a local work area directory and extracting an absolute path; constructing a logical chain skeleton according to parameters and the absolute path, compiling to generate a directed graph state machine; and encapsulating and packaging to push to a mixed reality end to perform space rendering. The application converts development intentions into machine instructions and outputs multi-modal assets, uses the absolute path to establish an addressing identifier to improve correlation accuracy, simultaneously establishes a bottom-layer hot replacement channel in a runtime environment, maps a file modification operation to a rendering pipeline across a compiling and packaging link, and shortens feedback time of scene iteration and rendering preview.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, specifically to an intention-driven method, system, and storage medium for automatically generating mixed reality content. Background Technology

[0002] With the development of mixed reality technology, the demand for mixed reality content development is increasing. In existing mixed reality content generation processes, developers typically need to create 3D physical models and spatial audio resources separately through distributed operations, and manually write business script code to implement the interaction logic. This approach fragments the digital content generation process, fails to translate the development intent in natural language form into low-level machine execution instructions, and suffers from low automation.

[0003] After the initial creation of multimodal assets is completed, it is necessary to bind the discrete assets with interactive events. The conventional approach relies on developers manually dragging and dropping resources in a visual editor. Due to the lack of a stable addressing mechanism based on underlying absolute paths, path references are easily lost or interactive logic breaks can occur when file hierarchy changes or cross-module calls are performed, making it difficult to guarantee the accuracy of the mapping association between multimodal assets and interactive logic nodes.

[0004] Furthermore, during the iterative phase of mixed reality scenarios, modifications to local resource files often require re-executing the entire project compilation and packaging process before being pushed to the edge rendering pipeline for display. The existing mixed reality runtime environment lacks a low-level hot-swap path across the compilation stage, resulting in lengthy scene iteration and rendering preview feedback times after modifications to files in the local workspace directory, thus reducing overall development efficiency. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intention-driven method, system, and storage medium for automatic generation of mixed reality content. It solves the problems in existing technologies, such as the reliance on manual resource import and logic script writing for mixed reality content development, the difficulty in automatically binding multimodal assets with interactive events, and the inability to synchronize local modifications to the edge rendering pipeline in real time.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides an intention-driven method, system, and storage medium for automatic generation of mixed reality content, comprising the following steps: The intent feature extraction and parsing module parses structured intent data and generates a machine-generated parameter set containing trigger parameters, prompt word features, and logical metadata. The multimodal asset collaborative generation module calls the multimodal large model matrix to process the machine-generated parameter set, generating a multimodal asset set containing the original 3D entity file, spatial audio file, and logical package metadata; The local workspace asset synchronization and hot replacement module reduces the original 3D entity file and outputs an optimized 3D entity file. The optimized 3D entity file and the spatial audio file are written to the local workspace directory, and the absolute paths of the optimized 3D entity file and the spatial audio file are extracted. The sentence-based visual logic binding module constructs a triplet logic chain skeleton based on the trigger parameters, the logic metadata, and the logic package metadata, writes the absolute path into the component binding mapping table, and compiles to generate a directed graph state machine. The mixed reality rendering and real-time preview module encapsulates the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, packages them into a compressed archive file, and pushes it to the mixed reality terminal to perform spatial rendering.

[0007] Preferably, the step of the intent feature extraction and parsing module parsing structured intent data and generating a machine-generated parameter set specifically includes: the intent feature extraction and parsing module converting intent data nodes in the structured intent data into intent feature vectors containing timeline sequence features, target description features, activity type features, and evaluation mechanism features; using a named entity recognition algorithm to extract entity nouns and attribute modifiers from the target description features and assembling them into the prompt word features; extracting interactive action identifiers from the activity type features to generate the trigger parameters as activation conditions for underlying state machine interactive events; converting the timeline sequence features into time delay parameters in the state machine execution flow, and converting the evaluation mechanism features into judgment thresholds for conditional branches in the underlying execution code; and merging the time delay parameters and the judgment thresholds to constitute the logical metadata.

[0008] Preferably, the hot-replacement step performed by the local workspace asset synchronization and hot-replacement module specifically includes: the local workspace asset synchronization and hot-replacement module starts a background listening process to call the file system application interface of the underlying operating system to capture file modification events generated in the local workspace directory; when the file modification event triggered by the replacement of an external resource file with the same name is captured, the external resource file with the same name is loaded into the running memory as new multimodal asset data according to the absolute path of the external resource file with the same name; the logical component set and spatial transformation matrix mounted on the original target scene entity in the scene node tree of the mixed reality rendering engine are extracted and cached; the new multimodal asset data is used to overwrite the mesh renderer data or audio stream data of the original target scene entity, and the cached logical component set and spatial transformation matrix are rebound to the updated scene entity.

[0009] Preferably, the step of constructing a triplet logic chain skeleton based on the trigger parameters, the logic metadata, and the logic package metadata, and compiling it into a directed graph state machine specifically includes: the sentence-based visual logic binding module constructing a visual view composed of a condition column panel carrying trigger condition nodes, a logic column panel carrying logic processing nodes, and an action column panel carrying execution action nodes, horizontally spliced ​​together; when an operation is detected that the optimized 3D entity file or the spatial audio file is assigned to the action column panel or the condition column panel, the sentence-based visual logic binding module penetrates the application layer to directly access the file system of the local operating system to obtain the absolute path of the optimized 3D entity file or the spatial audio file, and writes the absolute path into the corresponding execution action node or the trigger condition node; the sentence-based visual logic binding module converts the execution action node into an action executor through the underlying state machine compilation layer, maps the trigger condition node into an event listener component, and converts the logic processing node into a logic controller, outputting the directed graph state machine composed of the event listener component, the logic controller, and the action executor.

[0010] Preferably, the mixed reality rendering and real-time preview module encapsulates the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, packages them into a compressed archive file, and pushes it to the mixed reality end for spatial rendering. Specifically, this includes: the runtime engine of the mixed reality rendering and real-time preview module unpacking the received compressed archive file, reconstructing a scene node tree object containing hierarchical relationships and component binding mapping tables by deserializing the structured data dictionary; calling the underlying file input / output application interface according to the absolute path recorded in the component binding mapping table to load the optimized 3D entity file and the spatial audio file into the runtime memory of the mixed reality device; injecting the loaded optimized 3D entity file and the spatial audio file into the spatial rendering pipeline for spatial transformation and fragment shading calculations, and running the directed graph state machine concurrently with the concurrent rendering of graphics and audio.

[0011] Preferably, the intent feature extraction and parsing module uses a parameter parser to parse the intent feature vector to generate the machine-generated parameter set, uses a task scheduler to establish parallel asset generation tasks, and routes them to the multimodal large model matrix through an application programming interface; the multimodal asset collaborative generation module adds a two-channel output instruction and environmental reverberation parameters to the request header when performing audio generation to generate the spatial audio file carrying three-dimensional spatial positioning attributes.

[0012] Preferably, the sentence-based visual logic binding module calls and executes a cross-layer automatic binding algorithm at the underlying level to directly map the absolute path to the generated component binding mapping table.

[0013] Preferably, the mixed reality rendering and real-time preview module uses the global packaging module for executing resource archiving to call the serialization mapping function to convert the structured data dictionary in the runtime memory into a continuous byte stream, and merges it into the compressed archive file based on the absolute path; on the mixed reality end, the spatial audio mixing pipeline and graphics rendering pipeline based on the header-related transfer function are used to receive and concurrently render the optimized 3D entity file and the spatial audio file loaded based on the absolute path.

[0014] A second aspect of the present invention provides an intent-driven mixed reality content automatic generation system, comprising: The intent feature extraction and parsing module is used to parse structured intent data, generate a machine-generated parameter set containing trigger parameters, prompt word features and logical metadata, and execute tasks concurrently through the internally integrated artificial intelligence agent pipeline. The multimodal asset collaborative generation module is communicatively connected to the intent feature extraction and parsing module. It is used to call the multimodal large model matrix to process the machine-generated parameter set and generate a multimodal asset set containing the original 3D entity file, spatial audio file and logical package metadata. The local workspace asset synchronization and hot replacement module is communicatively connected to the multimodal asset collaborative generation module. It is used to reduce the dimensionality of the original 3D entity file and output the optimized 3D entity file, write the optimized 3D entity file and the spatial audio file into the local workspace directory, and extract the absolute paths of the optimized 3D entity file and the spatial audio file. The sentence-based visual logic binding module is communicatively connected to the intent feature extraction and parsing module and the local workspace asset synchronization and hot replacement module. It is used to construct a triplet logic chain skeleton based on the trigger parameters, the logic metadata and the logic package metadata, write the absolute path into the component binding mapping table, and compile to generate a directed graph state machine. The mixed reality rendering and real-time preview module is connected to the sentence-based visual logic binding module. It is used to encapsulate the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, package them into a compressed archive file, and push them to the mixed reality terminal for spatial rendering.

[0015] A third aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein a processor executes the computer program to implement the intention-driven mixed reality content automatic generation method as described in the first aspect of the present invention.

[0016] This invention provides an intent-driven method, system, and storage medium for automatic generation of mixed reality content. It offers the following advantages: 1. This invention generates a machine-generated parameter set containing trigger parameters and logical metadata by parsing structured intent data, and calls a multimodal large model matrix to process the parameter set to generate 3D entity files and spatial audio files. It transforms the development intent in natural language form into executable low-level machine instructions and outputs multimodal assets concurrently in the same running process. This solves the problem of relying on manual distribution to create 3D models and manually writing business scripts in traditional mixed reality scene development, and improves the automation level of digital content generation.

[0017] 2. This invention extracts the absolute paths of optimized 3D entity files and spatial audio files from the local workspace directory, uses a sentence-based visual logic binding module to construct a triplet logic chain skeleton based on trigger parameters and logic package metadata, and writes the absolute paths into the component binding mapping table. Then, it compiles and generates a directed graph state machine, using the underlying absolute path of the operating system as a unique addressing identifier to establish a direct mapping relationship between discrete multimodal assets and interactive logic nodes, avoiding the loss of path references or logic breakage caused by manually dragging and dropping resources.

[0018] 3. This invention captures file modification events in the local workspace directory by starting a background monitoring process. When an external resource file with the same name is detected to be replaced, the logical component set and spatial transformation matrix on the original target scene entity are extracted and cached. Then, the newly loaded multimodal asset data is used to overwrite the original rendering data and rebind the cached components. A hot replacement path based on the system's underlying monitoring mechanism is established in the mixed reality runtime environment, so that the modification operation of the local resource file can be directly mapped to the edge rendering pipeline across the compilation and packaging stage, shortening the feedback time of scene iteration and rendering preview. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system functional architecture according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 3 This is a schematic diagram comparing the resource addressing error rate under multiple hot reloads according to an embodiment of the present invention. Figure 4 This is a schematic diagram comparing the mixed reality rendering performance before and after system algorithm processing according to an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see the appendix Figure 1 This invention provides an intent-driven mixed reality content automatic generation system, which includes: an intent feature extraction and parsing module, a multimodal asset collaborative generation module, a local workspace asset synchronization and hot replacement module, a sentence-based visual logic binding module, and a mixed reality end-rendering and real-time preview module.

[0022] The intent feature extraction and parsing module constitutes the intent input and parsing layer, used to receive structured intent data input from the front-end configuration interface. This module performs feature extraction and semantic parsing on the structured intent data, converting natural language data and labeled features into a machine-generated parameter set (i.e., extracted and generated parameters) containing trigger parameters, prompt word features, and logical metadata, and then distributes the generated Prompt and parameters to the back-end pipeline.

[0023] The multimodal asset collaborative generation module communicates with the intent feature extraction and parsing module. As an AI agent collaborative generation layer, the multimodal asset collaborative generation module receives the machine-generated parameter set output by the intent feature extraction and parsing module. Based on the machine-generated parameter set, the multimodal asset collaborative generation module calls the multimodal large model matrix to generate multimodal assets in parallel. The multimodal assets include 3D entity files output from the 3D asset generation branch, spatial audio files output from the spatial audio TTS branch, and logical package metadata output from the pre-made logical package encapsulation branch. The generated multimodal assets are then distributed to the local system via a routing synchronization mechanism.

[0024] The local workspace asset synchronization and hot-replacement module communicates with the multimodal asset collaborative generation module. Together with the sentence-based visual logic binding module (described later), the local workspace and logic binding layer is formed. The local workspace asset synchronization and hot-replacement module receives the 3D entity files and spatial audio files output by the multimodal asset collaborative generation module via the aforementioned routing synchronization mechanism. The local workspace asset synchronization and hot-replacement module performs format verification and polygon reduction optimization on the received files, and then physically writes the processed files into the system's preset local workspace directory.

[0025] The local workspace asset synchronization and hot-swap module is also used to execute the directory monitoring and hot-swap mechanism, that is, to continuously monitor the file read and write status of the local workspace directory. When it is detected that a multimodal asset has been written or a file with the same name has been replaced, the local workspace asset synchronization and hot-swap module executes the scene entity replacement algorithm, maps the newly written assets to the resource pool of the mixed reality rendering engine, and retains the triggering events, logical conditions and action behavior data associated with the old entities.

[0026] The sentence-based visual logic binding module communicates with the intent feature extraction and parsing module, the multimodal asset collaborative generation module, and the local workspace asset synchronization and hot replacement module. Based on the trigger parameters output by the intent feature extraction and parsing module and the logic package metadata output by the multimodal asset collaborative generation module, the sentence-based visual logic binding module generates a triplet logic chain skeleton containing trigger conditions, logical processing, and execution actions.

[0027] The sentence-based visual logic binding module mounts assets synchronized with the local workspace asset synchronization and hot-replacement module to the corresponding nodes in the triplet logic chain skeleton, completing the binding of interactive logic (i.e., executing automatic asset mounting). The user interface displays the data and receives parameters in a three-column layout containing conditions, logic, and actions. The bound business data resources are then packaged and output, and transferred to the mixed reality rendering end via a hot-update mechanism.

[0028] The mixed reality edge rendering and real-time preview module communicates with the sentence-based visual logic binding module. The mixed reality edge rendering and real-time preview module constitutes the mixed reality rendering and real-time preview layer. It acquires scene nodes, multimodal assets, and logic chain data transmitted by the sentence-based visual logic binding module via the packaged output channel. The mixed reality edge rendering and real-time preview module encapsulates this data into a structured data dictionary. It then parses the structured data dictionary into a rendering tree in a spatial coordinate system and pushes it to the mixed reality headset or previewer for spatial rendering.

[0029] See attached document Figure 2 Based on the above system architecture, embodiments of the present invention also provide an intent-driven method for automatically generating mixed reality content, including the following steps: S1, the intent feature extraction and parsing module obtains the structured intent data input by the user, and performs intent feature extraction and intent semantic parsing to extract generation parameters and generate a machine-generated parameter set, and then distributes the generated Prompt and corresponding parameters.

[0030] S2, the multimodal asset collaborative generation module receives the machine-generated parameter set output by the intent feature extraction and parsing module, calls the multimodal large model matrix to generate assets, executes 3D asset generation, spatial audio TTS and pre-made logic package encapsulation in parallel, outputs three-dimensional entity files, spatial audio files and logic package metadata, and triggers routing synchronization instructions.

[0031] S3, the system performs local workspace synchronization and replacement. The local workspace asset synchronization and hot-replacement module receives the 3D entity files and spatial audio files generated by the multimodal asset collaborative generation module, performs formatting processing, and writes them to the local workspace directory. The local workspace asset synchronization and hot-replacement module monitors changes in the directory's status and achieves asset synchronization and replacement through directory monitoring and hot-replacement mechanisms and scene entity replacement algorithms.

[0032] S4, the system executes sentence-based visual logic binding. The sentence-based visual logic binding module generates a triplet logic chain skeleton based on the parameters parsed by the intent feature extraction and parsing module, and mounts the assets synchronized by the local workspace asset synchronization and hot-replacement module to this skeleton, completing the binding of visual logic and automatic asset mounting, and finally triggering the packaging output and hot-update operation.

[0033] S5 performs mixed reality rendering and real-time preview. The mixed reality rendering and real-time preview module packages and outputs the data assembled by the sentence-based visual logic binding module, converts it into a structured data dictionary, and pushes it to the mixed reality end for spatial rendering and preview.

[0034] See attached document Figure 1 The intent feature extraction and parsing module acquires structured intent data input by the user. The system front-end provides a visual intent configuration interface, which is built based on structured data models such as fishbone diagrams or tree topologies. Users input business logic or teaching logic in this visual intent configuration interface.

[0035] The intent feature extraction and parsing module receives intent data nodes from the user interface through a data collection mechanism. These intent data nodes contain sub-feature data across multiple dimensions. The system abstracts the collected intent data nodes into intent feature vectors. The intent feature vectors are formally represented using the following formula: ; In the formula: This represents the original input intent vector; Indicates the characteristics of a timeline sequence; Indicate the target's descriptive features; Indicates activity type characteristics; Indicates the characteristics of the evaluation mechanism; This indicates a transpose operation on a matrix or vector composed of characteristic elements.

[0036] Timeline sequence features are used to control the order in which different events and assets are triggered in a mixed reality scene. The system records the timestamp information of each event node through the timeline component in the visual intent configuration interface to generate timeline sequence features. For the rendering of the timeline component and the capture of timestamp data in the front-end graphical user interface, those skilled in the art can use conventional front-end development frameworks to implement it. The specific data acquisition implementation method is a well-known technology in this field and will not be described in detail here.

[0037] Target description features are used to record textual information about specific business tasks in mixed reality scenarios. The system receives natural language text entered by the user in the interface text boxes and stores it as target description features.

[0038] Activity type features define the interaction patterns within a scene. Specific sub-features of activity type features include observation type, interaction type, and quiz type. The system obtains the activity type feature selected by the user through preset label components in the interface. Observation type corresponds to static model display and audio playback behaviors in the scene. Interaction type corresponds to collision, grabbing, or raycasting behaviors between the user and 3D entities. Quiz type corresponds to the system outputting questions to the user and waiting for input.

[0039] The evaluation mechanism features define the rules for judging the results of user-space operations. The system receives numerical thresholds or logical judgment conditions set by the user and uses them as evaluation mechanism features. These features are used in conjunction with the test types mentioned above in the activity type features to establish conditional branches in the underlying state machine.

[0040] The intent feature extraction and parsing module transforms the discrete configuration input from the front end into intent feature vectors containing the four defined dimensions mentioned above, thus completing the conversion from unstructured intent to a standardized data structure.

[0041] The intent feature extraction and parsing module receives the aforementioned intent feature vector. The system then uses a pre-defined intent semantic parser to perform semantic parsing on the natural language data and labeled features in the intent feature vector.

[0042] The intent semantic parser transforms the intent feature vector into a machine-generated parameter set through a semantic parsing mapping function. The formal transformation formula is as follows: ; In the formula: This represents the machine-generated parameter set generated after parsing; This represents the semantic parsing mapping function applied to the original input intent vector; Indicates the trigger parameters; Indicates the characteristics of the prompt words; This represents the logical metadata characteristics. The system performs targeted parsing operations on the different sub-feature data contained in the intent feature vector to generate the three independent machine-generated parameters mentioned above.

[0043] The system parses the activity type feature in the intent feature vector and maps it to trigger parameters. These trigger parameters control the activation conditions of interactive events in the underlying state machine. When the lower-level feature of the activity type feature is an interaction type or a quiz type, the semantic parsing mapping function extracts the interactive action identifier and generates trigger parameters for the corresponding mixed reality physics collision component or ray detection component. The specific calling and binding logic of the collider component and ray detection component in the underlying physics engine can be implemented using existing 3D rendering engine interfaces by those skilled in the art; the implementation method of its underlying interaction detection is well-known in the field and will not be elaborated here.

[0044] The system parses the target description features in the intent feature vector and maps them to cue word features. Cue word features are the text input conditions used to drive subsequent multimodal model generation of 3D model assets or audio assets. The intent semantic parser uses a named entity recognition algorithm to extract specific entity nouns and attribute modifiers from the natural language text of the target description features. The system concatenates the extracted entity nouns and attribute modifiers according to a preset data template, assembling them into cue word features that conform to the input specifications of generative large models. When the target description features involve a structural cognition task of a specific instrument, the intent semantic parser extracts the name of the instrument and its structural feature parameters, integrating them into the cue word features.

[0045] The system combines the timeline sequence features and evaluation mechanism features from the intent feature vector to generate logical metadata features. These logical metadata features define the behavior patterns and feedback mechanisms of 3D model assets in specific mixed reality scenarios. The intent semantic parser transforms the timeline sequence features into time delay parameters in the state machine execution flow and the evaluation mechanism features into judgment thresholds for conditional branches in the underlying execution code. The system merges and packages the time delay parameters and judgment thresholds to form complete logical metadata features.

[0046] The intent feature extraction and parsing module reduces the dimensionality of the human-readable abstract intent feature vector through the above parsing operations and maps it into a machine-generated parameter set that can be directly executed by the underlying hardware and algorithms. The machine-generated parameter set is then output to the multimodal asset collaborative generation module.

[0047] The multimodal asset collaborative generation module receives the machine-generated parameter set output by the aforementioned parsing. Internally, the multimodal asset collaborative generation module includes an AI agent pipeline, which is used to distribute and control the parallel invocation of the received machine-generated parameter set.

[0048] The AI ​​agent pipeline includes a parameter resolver and a task scheduler. The parameter resolver reads various feature data from the machine-generated parameter set. The task scheduler establishes parallel asset generation tasks based on the type of feature data. The system routes and distributes the concurrent generation tasks to the corresponding multimodal AI large model matrix.

[0049] The multimodal AI large model matrix includes a 3D generative model for generating visual entities, a text-to-speech model for generating spatial audio, and a logic generation engine for encapsulating execution scripts.

[0050] Specifically, after acquiring the features of the prompt words, the AI ​​agent pipeline uses them as input conditions and sends them to the 3D generation model and the text-to-speech model via the application programming interface. The multimodal asset collaborative generation module, after acquiring the logical metadata features, routes them to the logical generation engine.

[0051] The system maintains the parallel execution of the aforementioned multiple generation tasks through multithreading or asynchronous coroutine technology. After the processing results are returned from each AI model interface, the task scheduler performs unified aggregation and verification of the returned data. This verification process includes: reading the file extensions of the returned 3D entity files to verify whether they conform to the preset 3D format requirements, reading the file header information of the spatial audio files to verify the integrity of the audio data, and checking whether the logical package metadata contains complete execution node fields. The multimodal asset collaborative generation module outputs a multimodal asset set containing 3D entity files, spatial audio files, and logical package metadata. The parallel generation result of the multimodal asset set is formally represented by the following formula: ; In the formula: This represents the final collection of multimodal assets generated by the system. This represents the original 3D solid file output by the 3D generated model in the multimodal artificial intelligence large model matrix; This represents the spatial audio file output by the text-to-speech model in a multimodal artificial intelligence large model matrix. This represents the logical package metadata output by the logic generation engine based on the characteristics of the logical metadata.

[0052] For the network communication mechanism between the AI ​​agent pipeline and each underlying AI model based on the application programming interface, those skilled in the art can use conventional network transmission protocols and load balancing techniques to implement it. The underlying network request and concurrency control implementation methods are well-known technologies in the field and will not be elaborated here.

[0053] The AI ​​agent pipeline uses the aforementioned distribution routing and parallel invocation mechanism to transform the machine-generated parameter set output by the intent feature extraction and parsing module into the 3D visual, auditory, and logical data required for the mixed reality scene and then outputs it.

[0054] The multimodal asset collaborative generation module utilizes the 3D entity generation task and spatial audio generation task distributed by the aforementioned task scheduler to perform specific multimodal asset generation operations.

[0055] The multimodal asset collaborative generation module extracts cue word features from the machine-generated parameter set to generate visual model assets. These cue word features contain structured text that explicitly describes the geometry, surface material properties, and polygon count limitations of the 3D objects required for the mixed reality scene. The system encapsulates the cue word features containing these attributes into a data request payload and sends it to the 3D generative model via an application programming interface (API).

[0056] The 3D generated model performs visual calculations and mesh reconstruction based on the features of the received prompt words, outputting the original 3D solid file. The generation process of the original 3D solid file is formally represented by the following formula: ; In the formula: This refers to the original 3D solid file output from the 3D generated model. This represents the mesh generation mapping function corresponding to the 3D generated model; Features of input prompt words.

[0057] The original 3D solid file uses a common 3D model data format in the field of computer graphics, specifically including OBJ, FBX, or GLTF formats. The internal data structure of the original 3D solid file contains vertex coordinate data, normal vector data, and texture mapping coordinate data for material mapping of the 3D mesh.

[0058] The multimodal asset collaborative generation module simultaneously processes the spatial audio generation task while performing the 3D entity generation task. The system parses the speech text to be converted, emotion parameters, and speech rate parameters from the cue word features. Specific sub-features of the emotion parameters include calm, excited, or questioning indicators to control the tone of voice. The speech rate parameter includes a numerical indicator of the number of words read per minute to control the rhythm of speech.

[0059] The system integrates the spoken text, emotion parameters, and speech rate parameters into the text-to-speech model engine. The engine combines the emotion and speech rate parameters to calculate acoustic features and synthesize waveforms from the spoken text, outputting a spatial audio file. The generation process of the spatial audio file is formally represented by the following formula: ; In the formula: This indicates the generated spatial audio file; This represents the acoustic synthesis mapping function from text to the speech model engine; This represents cue word features that include spoken text, emotion parameters, and speech rate parameters.

[0060] The output spatial audio file is specifically formatted as WAV or MP3. To ensure the generated audio file meets the spatial sound requirements of a mixed reality environment, the multimodal asset collaborative generation module appends a two-channel output command and preset environmental reverberation parameters to the request header when sending a request to the text-to-speech model engine. By introducing these two-channel output commands and environmental reverberation parameters, the system ensures that the final generated spatial audio file's data stream carries three-dimensional spatial positioning attributes, enabling distance- and orientation-based audio attenuation and location determination within the mixed reality headset.

[0061] For neural radiation field or three-dimensional Gaussian diffusion algorithms based on text-to-3D conversion in 3D generative models, and acoustic waveform synthesis algorithms based on deep neural networks in text-to-speech model engines, those skilled in the art can implement them using existing open-source generative artificial intelligence large models. Their underlying model training and tensor inference mechanisms are well-known technologies in the field and will not be elaborated here.

[0062] While performing 3D model and spatial audio generation tasks, the multimodal asset collaborative generation module also processes the encapsulation of pre-made logic packages through its built-in logic generation engine. The logic generation engine receives and parses the logical metadata features and trigger parameters from the aforementioned generated machine generation parameter set, transforming them into pre-made logic packages that can be read by the underlying state machine.

[0063] A prefabricated logic package is a collection of variables and execution flows required for a specific interaction. The system structurally assembles scattered logic parameters and outputs prefabricated logic package metadata. The data structure of the prefabricated logic package metadata is formally represented by the following formula: ; In the formula: This represents the metadata of the prefabricated logic package; This represents a set of state variables used to record temporary data during mixed reality interactions. This represents a set of state transition rules, used to define the update logic for state variables after a specific event is triggered. This represents a set of conditional branches, used to record the judgment conditions for logical flow.

[0064] The logic generation engine executes differentiated encapsulation logic for different activity type characteristics. When the current intent data is identified as involving a quiz-type activity, the logic generation engine instantiates a specific set of state variables. The specific sub-characteristics of this set of state variables include a score counter variable for recording user action scores, a retry counter variable for recording the number of actions, and a timer variable for controlling the answering time.

[0065] The system extracts the judgment threshold parameters from the logical metadata features and maps them to the conditional branch set. In the quiz activity scenario, the conditional branch set contains the threshold comparison logic for the score counter variable or timer variable. The state transition rule set is bound to specific trigger parameters. When a preset physical collision or ray-click event occurs in the mixed reality scene, the state transition rule set drives the score counter variable to increment or drives the timer variable to stop updating.

[0066] The logic generation engine also encapsulates the time delay parameter in the logical metadata features into a pre-built logic package. The time delay parameter is compiled into asynchronous wait instructions in the set of state transition rules. When the mixed reality terminal performs spatial audio playback or model animation playback actions, this asynchronous wait instruction forces the underlying state machine to suspend for a specified period of time to ensure that the display duration of the multimodal assets is consistent with the timeline sequence features of the business intent.

[0067] The multimodal asset collaborative generation module outputs the assembled pre-built logic package metadata as a standardized Extensible Markup Language (EXPLAIN) file or a JavaScript object musical score file. This file serves as the data base for subsequent interaction logic binding, and together with the 3D entity file and spatial audio file, constitutes a complete multimodal asset set.

[0068] For the parameter extraction, node assembly, and data serialization and deserialization processes of the underlying logic generation engine based on the abstract syntax tree, those skilled in the art can use conventional script parsing frameworks and data structure serialization libraries to implement them. The underlying syntax tree construction and formatted output are well-known technologies in the field and will not be described in detail here.

[0069] The local workspace asset synchronization and hot-swap module receives a multimodal asset set output by the multimodal asset collaborative generation module. The multimodal asset set includes the original 3D entity files, spatial audio files, and logical package metadata.

[0070] Before physically writing assets to the local disk, the local workspace asset synchronization and hot-swap module performs format verification and polygon reduction optimization on the original 3D entity files. Because the graphics rendering performance of mixed reality devices is limited by the computing power of mobile chips, the system needs to control the polygon count of 3D models within the scene. The system applies a preset 3D mesh processing algorithm to perform mesh dimensionality reduction on the original 3D entity files. The specific features of this mesh dimensionality reduction operation include a polygon reduction mechanism based on edge folding algorithms or vertex clustering algorithms. The polygon reduction optimization process of the 3D entity is formally represented by the following formula: ; In the formula: This represents the 3D solid file output after polygon reduction optimization. This represents the system's preset grid dimensionality reduction mapping function; This represents the original 3D solid file that was input. This represents the maximum polygon count threshold preset by the system based on the performance of the mixed reality device. When the number of faces in the original 3D solid file exceeds the maximum polygon count threshold, the mesh dimensionality reduction mapping function removes redundant vertices and reconstructs the topology of the 3D mesh, outputting an optimized 3D solid file that conforms to rendering constraints.

[0071] For spatial audio files and logical packet metadata, the system performs file format extension matching and data integrity verification, and intercepts generated files that do not conform to the preset extension specifications or are corrupted.

[0072] The local workspace asset synchronization and hot-swap module writes the processed 3D entity files, spatial audio files, and logical package metadata to the system's preset local workspace directory. The system employs a single-project, single-directory resource management architecture to construct the local workspace. The path of the local workspace directory contains the project's unique identifier for the corresponding generation task. The system archives and stores multimodal assets generated under the same intent in an independent physical folder corresponding to the project's unique identifier. This file storage mechanism ensures that resource files between different generation tasks are isolated from each other, avoiding resource naming conflicts.

[0073] The local workspace asset synchronization and hot-swap module starts a background monitoring process for the local workspace directory. This background monitoring process continuously monitors the file read / write status of the local workspace directory. The system calls the underlying operating system's file system application programming interface (API) to obtain state change events within the local workspace directory. The specific characteristics of these state change events include file creation events, file modification events, and file deletion events.

[0074] Once the generated multimodal assets have been transmitted over the network and written to the local disk, or when a user drags and drops an external resource file with the same name into the directory, triggering a file modification event, the background monitoring process immediately captures the corresponding state change event and records the absolute path of the resource file that triggered the event, as well as the timestamp data. The local workspace asset synchronization and hot-swap module loads the new resource file into the runtime memory based on the obtained absolute file path, serving as input data for subsequent resource allocation.

[0075] For the call to the file system application interface and the thread management mechanism of the background listening process, those skilled in the art can use the file system observer service provided by the underlying operating system. The specific implementation method of file status capture is a well-known technology in the field and will not be described in detail here.

[0076] The local workspace asset synchronization and hot-replacement module constructs an underlying data synchronization channel between the cloud-generated model output data and the local mixed reality rendering engine through the aforementioned formatted disk writing and status monitoring mechanism, providing a pre-trigger condition for the dynamic iteration of scene resources in the system.

[0077] After the local workspace asset synchronization and hot replacement module detects changes in the file read / write status of the local workspace directory and captures file change events, it executes the scene entity replacement algorithm.

[0078] The local workspace asset synchronization and hot-swap module obtains the absolute path of the new asset file that triggered the state change. The system uses only absolute paths when reading and loading underlying physical files to prevent relative paths from causing file addressing failures or path resolution errors in multi-level project directory structures. Based on this absolute path, the local workspace asset synchronization and hot-swap module loads the newly generated 3D entity file or spatial audio file into the mixed reality rendering engine's runtime memory.

[0079] The local workspace asset synchronization and hot-replacement module retrieves the original target scene entity associated with the absolute path in the scene node tree of the mixed reality rendering engine. Before performing visual or auditory asset replacement, the system performs data decoupling and extraction operations on the original target scene entity. The local workspace asset synchronization and hot-replacement module extracts and caches the set of logical components and spatial transformation matrix attached to the original target scene entity.

[0080] The specific sub-features of the aforementioned logical component set include trigger event listeners, collider component parameters, and associated action behavior script data. The specific sub-features of the aforementioned spatial transformation matrix include the 3D coordinate parameters, Euler angle rotation parameters, and three-axis scaling parameters of scene entities in the world or local coordinate system. The system stores the extracted core logic and spatial parameters in a temporary data stack in memory. The execution process of the scene entity replacement algorithm is formally represented by the following formula: ; In the formula: This represents a completely new scene entity generated after the replacement operation is completed; This represents the scene entity replacement mapping function executed by the system; Represents the original target scene entities in a mixed reality scene; This indicates a new multimodal asset loaded from the local workspace via an absolute path; Represents the spatial transformation matrix of the extracted and cached original target scene entities; This represents the collection of logical components from the original target scene entities that have been extracted and cached.

[0081] The local workspace asset synchronization and hot-swap module uses the new multimodal asset data in memory to overwrite the original target scene entity's mesh renderer data or audio stream data. The system then reads the temporary data stack, remaps the cached spatial transformation matrix and logical component set, and binds them to the scene entity that has loaded the new assets.

[0082] Through the aforementioned decoupling and rebinding mechanisms, the system achieves seamless switching of underlying asset data. After asset replacement, new scene entities directly inherit the physical boundaries, event response logic, and spatial location of the old entities, ensuring the continuity and stability of interactive business logic in mixed reality scenarios.

[0083] For the scene node tree traversal and retrieval algorithm, the memory loading and rewriting mechanism of mesh data and audio data, and the memory garbage collection operation of old assets in the mixed reality rendering engine, those skilled in the art can call the conventional rendering engine application programming interface to implement them. The underlying memory management and node tree update operation are well known technologies in the field and will not be described in detail here.

[0084] After obtaining the trigger parameters and logic package metadata generated at the underlying level, the sentence-based visual logic binding module constructs a visual view for front-end interaction configuration. This view is the UI interaction interface layer shown in the attached diagram. The design of this UI interaction interface layer conforms to natural language habits and is generated based on the triple logic chain skeleton. In the user interface, it displays data and receives parameters in a three-column layout containing conditions (When), logic (How), and actions (Then).

[0085] The sentence-based visual logic binding module draws the condition column panel through the interface rendering engine. The condition column panel is used to hold trigger condition nodes. The specific sub-features of the trigger condition nodes include physical collision events in the mixed reality scene (such as the detected touch event shown in the attached figure), gaze events, gesture pinch events, and spatial ray click events. The system receives the trigger parameters set by the user through the drop-down selection component and input box in the condition column panel.

[0086] The sentence-based visual logic binding module draws a logic column panel horizontally connected to the condition column panel. This logic column panel hosts logic processing nodes. The specific features of each logic processing node include numerical comparison operations, Boolean logic operations, and time control logic (such as the 2-second delay trigger operation shown in the attached diagram). Numerical comparison operations cover greater than, equal to, and less than comparison rules. Boolean logic operations cover AND, OR, and NOT relationships between multiple trigger events. The system parses and populates the logic metadata output by the multimodal asset collaborative generation module into this logic column panel for users to perform secondary parameter tuning.

[0087] The sentence-based visual logic binding module draws the action column panel at the end of the view. The action column panel is used to hold action execution nodes. Specific sub-features of these action execution nodes include playing spatial audio, triggering 3D entity animation (as shown in the attached diagram), updating interface text variables, and switching spatial scenes. The triplet logic structure in the view hierarchy is formally represented by the following formula: ; In the formula: This represents the final visual view generated by the sentence-based visual logic binding module. This represents the system's preset interface layout mapping function; This indicates the trigger condition node rendered in the Conditions panel; This represents the logic processing node rendered in the logic column panel; This indicates the execution action node rendered in the Actions panel.

[0088] The system arranges the data nodes in the three panels horizontally on the front-end interface, forming a reading flow similar to natural language sentences, thus transforming complex code logic into easily readable sentence-like expressions.

[0089] The sentence-based visual logic binding module receives user resource binding operations within the interface. When a user assigns a local asset to the action or condition panel, the system extracts and records the absolute path of that asset on the local physical disk. This absolute path is stored as a unique resource pointer in the corresponding action node or trigger condition node. Using absolute path addressing avoids file location failures in multi-level project structures, ensuring that the underlying rendering engine can accurately load the corresponding physical entity file or audio file based on the absolute path when executing an action.

[0090] For the implementation mechanism of drag-and-drop interactive components in the front-end user interface and the two-way binding and update process of interface data, those skilled in the art can call existing responsive web front-end development frameworks to implement it. The specific user interface component rendering and lifecycle management are well-known technologies in this field and will not be elaborated here.

[0091] After the sentence-based visual logic binding module completes the parameter configuration of the triple logic chain in the front-end visual view, it triggers the underlying state machine compilation layer to perform cross-layer automatic binding and code compilation operations.

[0092] The underlying state machine compilation layer reads node data from the sentence-based visual logic view and serializes it into an intermediate logic object. This intermediate logic object fully records the data dependencies and hierarchical relationships of trigger condition nodes, logic processing nodes, and execution action nodes. The cross-layer automatic binding algorithm receives this intermediate logic object and performs the mapping and binding between the front-end business logic and the underlying physical assets.

[0093] The cross-layer automatic binding algorithm parses the execution action nodes and trigger condition nodes in the intermediate logic object, extracting the absolute paths of the asset files recorded therein. Based on these absolute paths, the system penetrates the application layer to directly access the local operating system's file system, locating the 3D entity file or spatial audio file in the local workspace directory. The system uses the extracted absolute path as a pointer and writes it into the generated component binding mapping table. The process of establishing this component binding mapping table implements the UI parameter mapping operation shown in the attached diagram, establishing a communication path from the upper-layer panel data to the lower-layer compiled components.

[0094] The underlying state machine compilation layer generates a directed graph state machine that the mixed reality engine can directly execute, based on the intermediate logic objects and component binding mapping table. Referring to the visual logic execution flow in the attached diagram, this directed graph state machine is composed of EventListener, Logic Controller, and Action Executor connected sequentially. The compilation and transformation process of the directed graph state machine is formally represented by the following formula: ; In the formula: This represents the directed graph state machine generated during compilation; This represents the compiler mapping function executed by the underlying state machine compiler layer; This represents the triplet logical structure data extracted from the sentence-based visual logical view; This represents the set of absolute paths of multimodal assets within the local physical disk; Represents a set of state nodes; Represents the set of edges that transition between states; This represents the generated component binding mapping table.

[0095] The underlying state machine compilation layer transforms the execution action nodes in the intermediate logic objects into a set of state nodes, which is instantiated as the Action Executor component in the underlying state machine compilation layer. Specific sub-characteristics of the state node set include audio playback state, animation execution state, and scene transition state. The system maps trigger condition nodes to the Event Listener component in the underlying layer and transforms logic processing nodes into a set of state transition edges (i.e., mapped to the Logic Controller component). The set of state transition edges defines the directed flow paths and activation thresholds between nodes within the state node set. Through the aforementioned UI parameter mapping mechanism, the system accurately transforms the front-end's detection of touch events, delayed triggering (2 seconds), and the concatenation configuration of playback object animation / audio into a compilation layer control flow where the underlying event listener sends signals to the logic controller, and the logic controller then triggers the action executor.

[0096] During compilation, the system injects the component binding map into the initialization function of the directed graph state machine. When the mixed reality scene starts, the underlying rendering engine reads the absolute paths in the component binding map, dynamically loads the corresponding 3D entity files or spatial audio files using the absolute paths, and mounts them to the corresponding state nodes. By forcing the use of absolute paths for resource binding, the system avoids asset loss issues caused by runtime errors in relative path hierarchy.

[0097] The compiled directed graph state machine is serialized into a script file natively supported by the mixed reality engine. The specific format of the native script file includes C# script components or C++ class files. The mixed reality rendering engine attaches this script file to the scene root node or a specific 3D entity, taking over the low-level interaction loop of the mixed reality environment.

[0098] For the node traversal algorithm based on directed graph in the underlying state machine, the lexical analysis and abstract syntax tree construction of the scripting language, and the reflection and dynamic component mounting mechanism at the runtime of the mixed reality engine, those skilled in the art can use conventional compiler front-end technology and rendering engine interface to implement them. The underlying script compilation and execution methods are well-known technologies in this field and will not be elaborated here.

[0099] Between the system's front-end stylized visual configuration and the underlying state machine compilation, there exists a triplet logic model serving as a data intermediary layer. Based on this triplet logic model, the stylized visual logic binding module structurally decomposes and reassembles the non-linear interaction processes in mixed reality scenarios.

[0100] The triplet logic model consists of a set of triggering conditions, a set of logical processing, and a set of execution actions. The system abstracts and encapsulates the data from these three sets into a tree-like logical chain data structure to facilitate parameter passing between modules.

[0101] For the specific structure of the triplet logic model, the system defines the corresponding lower-level feature implementations. The trigger condition set corresponds to the input events captured by the front end. The lower-level features of the trigger condition set include a spatial coordinate intersection calculator based on the spatial anchor point of the mixed reality device, a gaze ray collision detector based on eye-tracking hardware, and a voice command string matcher based on the microphone input stream.

[0102] The logic processing set is responsible for performing intermediate state operations on the state values ​​of the trigger conditions. The lower-level features of the logic processing set include AND gates, OR gates, Boolean NOT components for processing multiple combinations of trigger conditions, and a numerical interval comparator for determining spatial distance or user score.

[0103] The action set receives the results of logical processing and outputs instructions to the rendering engine. The underlying characteristics of the action set include calling the audio playback interface of the underlying rendering engine, triggering the model animation controller, and instruction modules that read and load scene resources via absolute paths. The data structure of the triplet logic model is formally represented by the following formula: ; In the formula: This represents the encapsulated triplet logical model data packet; Represents the set of trigger condition nodes; This represents a set of logical processing nodes; Represents the set of nodes that perform actions; This represents a tuple structure in which the three sets mentioned above are assembled in an ordered manner according to the execution flow.

[0104] Mixed reality scenes maintain a global state vector that records scene attributes at runtime. During each frame's graphics rendering cycle, the system calls event listeners to poll for data changes in the set of trigger condition nodes.

[0105] When a specific node in the trigger condition node set is activated by external interaction, the system passes the output status value of that node to the logic processing node set for Boolean algebra operations or numerical comparison operations.

[0106] The logic processing node set outputs the calculation result. The system determines whether to block the execution flow based on the calculation result. If the calculation result meets the preset activation conditions, the system sends an activation signal to the action node set, instructing the underlying engine to execute the specific presentation layer action. The underlying logic operation and state update process of the system is formally represented by the following formula: ; In the formula: This represents the calculated global state vector of the mixed reality scene in the next frame: This represents the pre-defined logic execution mapping function within the underlying state machine: The scene global state vector representing the current frame: This represents a triplet logical model data packet.

[0107] When executing action node sets to invoke specific visual or auditory assets, the system strictly reads the absolute path of the local physical disk recorded in the node attributes. The system uses this absolute path as the sole input parameter to the underlying file stream reading function, skipping the mixed reality engine's default relative path resource lookup logic. This mechanism allows the system to directly access the underlying operating system's file nodes, preventing runtime resource loading failures caused by changes in the project directory structure or errors in relative path hierarchy.

[0108] For the acquisition of underlying sensor spatial data of mixed reality devices, the frame update loop mechanism of graphics rendering engine, and the pointer traversal algorithm of tree data structure in memory, those skilled in the art can use the mixed reality development kit interface provided by various hardware manufacturers to implement them. The underlying hardware driver call and basic algorithm traversal method are well known technologies in this field and will not be described in detail here.

[0109] By constructing the aforementioned triplet logical model, the system achieves data isomorphism between business intent, front-end view, and underlying execution code, ensuring data consistency and traceability of logical nodes during cross-level transmission.

[0110] After completing the compilation of the underlying state machine and automatic binding across layers, the system performs the assembly and serialization output of the structured data dictionary through the global packaging module.

[0111] The global packaging module collects index information of the multimodal asset set in runtime memory, pre-built logic package metadata, and the compiled directed graph state machine object. The system integrates these discrete data units into a unified structured data dictionary. The structured data dictionary is constructed using a key-value pair mapping data structure. In this data structure, the key field stores a universally unique identifier for each entity node in the mixed reality scene, and the value field encapsulates the spatial transformation matrix of the corresponding entity node, the bound logic script pointer, and the read address of the physical asset file. The generation and serialization process of the structured data dictionary is formally represented by the following formula: ; In the formula: This represents the global configuration text output after serialization. This represents the serialization mapping function called by the globally packaged module; This represents the scene node hierarchy and spatial coordinate metadata; This represents the set of absolute paths of multimodal assets within the local physical disk; This indicates the logical binding relationship between the underlying components and scene entities.

[0112] The global packaging module calls the serialization mapping function to convert the structured data dictionary object tree stored in runtime memory into a continuous byte stream or formatted text. Specific features of this serialization operation include converting memory objects into JavaScript object notation files, Extensible Markup Language (XML) files, or Protocol Buffers (PBR) binary files. This global configuration text serves as a static snapshot of the entire mixed reality scene, recording all parameters and dependencies required for scene initialization.

[0113] After the configuration text is serialized, the global packaging module performs the archiving and packaging operation of the physical resource files. The system reads the set of absolute paths recorded in the global configuration text. Based on this set of absolute paths, the system penetrates the underlying operating system application programming interface to locate and read the 3D entity files and spatial audio files in the local workspace directory. The system forces the use of absolute paths for file addressing, avoiding file omission errors caused by root directory drift when the packaging tool resolves relative paths.

[0114] The system combines the read physical asset files with the serialized global configuration text into a single compressed archive file. Specific features of this archiving operation include generating independent distribution packages with specific file extensions using ZIP compression algorithms or the TAR packaging specification. This compressed archive file, as the final carrier of the mixed reality content, can be directly transmitted to the head-mounted display device for loading and rendering.

[0115] For key-value pair traversal algorithms for memory objects, recursive serialization parsing of tree data structures, and compression encoding of underlying file system byte streams, those skilled in the art can call existing standard data processing libraries and compression algorithm libraries to implement them. The underlying data encoding and file packaging mechanisms are well-known technologies in this field and will not be elaborated here.

[0116] The mixed reality device receives the compressed archive file output by the aforementioned global packaging module. The runtime engine of the mixed reality device performs an unpacking operation on the compressed archive file and extracts it to the specified workspace directory in the device's local storage sandbox.

[0117] The runtime engine reads the unpacked global configuration text. The system then uses deserialization to parse the global configuration text into a scene node tree object in memory. This scene node tree object fully reconstructs the hierarchical relationships, spatial transformation matrices, and component binding mapping tables recorded in the structured data dictionary.

[0118] After reconstructing the scene node tree, the runtime engine performs the instantiation and loading of physical assets. Based on the root node of the specified workspace directory and the resource identifiers recorded in the global configuration text, the system constructs the absolute paths of each physical asset file in the mixed reality device's local file system. The runtime engine strictly relies on these absolute paths, calling the underlying file input / output application programming interfaces to load the 3D entity files and spatial audio files into runtime memory or video memory. By forcing the use of absolute paths for file resource addressing during device runtime, the system avoids asset loading blocking issues caused by working directory drift or relative path resolution errors in mobile sandbox operating systems.

[0119] The loaded multimodal assets are injected into the mixed reality spatial rendering pipeline. The spatial rendering pipeline, combined with real-time acquired device pose data, outputs a sequence of multimodal frames for user viewing and listening. The calculation process of this rendering pipeline is formally represented by the following formula: ; In the formula: This refers to the rendering results, including stereoscopic images and spatial audio, output by the spatial rendering pipeline to the device screen and speakers. This indicates the spatial rendering pipeline mapping function called by the underlying rendering engine; This represents the spatial pose matrix of the head-mounted display device, obtained in real time from hardware sensors. This represents the scene node tree object generated by deserialization; This represents a collection of multimodal assets loaded into memory via an absolute path.

[0120] The specific features of the spatial rendering pipeline include a 3D graphics rendering pipeline based on rasterization calculations, and a spatial audio mixing pipeline based on head-related transfer functions. The system submits the vertex and texture data of the 3D solid file to the graphics rendering pipeline for coordinate space transformation and fragment shading calculations, while simultaneously submitting the waveform data of the spatial audio file to the audio mixing pipeline for spatial acoustic calculations based on physical distance attenuation and sound image orientation.

[0121] While graphics and audio are rendered concurrently, the mixed reality device starts and runs the underlying directed graph state machine. Based on the component binding mapping table, the system attaches logic script instances to the corresponding scene nodes. In each frame's rendering loop, the logic script continuously polls the hardware input interface of the mixed reality device to obtain spatial ray intersections, gesture recognition coordinates, and gaze tracking vectors. When the aforementioned hardware input data meets the trigger condition thresholds defined in the logic script, the state machine executes node state transition operations, driving related entities in the scene node tree to execute preset model animation playback or sound response commands.

[0122] For the spatial pose calculation based on synchronous positioning and mapping algorithms, the shader calling mechanism of the graphics rendering application programming interface, and the underlying algorithms of head-related transfer functions in audio spatialization, those skilled in the art can use existing general rendering engine architectures and mixed reality underlying driver libraries to implement them. The underlying matrix coordinate operations and hardware communication protocols are well-known technologies in the field and will not be elaborated here.

[0123] The mixed reality terminal device uses the aforementioned data parsing, absolute path resource loading, and pipeline rendering mechanisms to transform the generated static data packets into dynamic spatial content that responds to interactive actions in real time.

[0124] Electronic devices are used to carry and run the aforementioned intent-driven mixed reality content automatic generation system. The hardware architecture of the electronic devices includes processors, memory, and communication interfaces that are interconnected via a system bus.

[0125] As the control and computational center of electronic devices, the processor is responsible for calling and executing various computational and logical tasks within the system. Specific features of a processor include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor, or a neural network processor. In parallel inference operations involving large multimodal artificial intelligence model matrices, the system schedules the GPU or tensor processor to perform the underlying matrix multiplication and addition calculations.

[0126] Memory is used to store computer program instructions that can be executed by the processor, as well as physical files generated during operation. Specific characteristics of memory include random access memory, read-only memory, electrically erasable programmable read-only memory, or solid-state drive. The aforementioned machine-generated parameter set, multimodal asset set, and structured data dictionary all reside as business data in the allocated addresses of this memory.

[0127] The communication interface is used to handle data network interactions between electronic devices and external mixed reality headsets or cloud servers. Specific features of the communication interface include Ethernet network interfaces, wireless LAN transceiver modules, or cellular mobile communication baseband chips.

[0128] The system's operation in electronic devices manifests as the reading and writing of low-level data and the execution of computational instructions. The processor receives externally input intent data via a communication interface and stores it in the input buffer area of ​​memory. The processor then retrieves the code instructions from memory to perform intent parsing, multimodal asset collaborative generation, and low-level state machine compilation.

[0129] The processor, based on the control logic of the code instructions, performs parameter extraction and AI model interface call operations on the intent data in the cache area. The processor generates the aforementioned execution action nodes and trigger condition nodes containing absolute paths in the running memory space. The processor writes the dimensionality-reduced optimized 3D entity file, spatial audio file, and serialized global configuration text to the local working directory of the memory, completing the scene resource write-to-disk operation. The hardware execution process of the electronic device is formally represented by the following formula: ; In the formula: This refers to a compressed archive file data stream that an electronic device outputs to the outside via a communication interface. A function representing the arithmetic mapping of the processor's underlying instruction cycle; This indicates that mixed reality content residing in memory automatically generates program instructions; This indicates that the initial content received through the communication interface generates request data; This represents the data transmission bandwidth control parameter of the system bus.

[0130] This invention provides a computer-readable storage medium. A computer program is recorded on this computer-readable storage medium. The aforementioned functional modules, such as the intent feature extraction and parsing module, the multimodal asset collaborative generation module, and the sentence-based visual logic binding module, are physically equivalent to binary machine code recorded in specific sectors of this computer-readable storage medium. When the computer program is read and executed by a processor, it implements the aforementioned processing and packaging procedures for the automatic generation of mixed reality content. Specific features of the computer-readable storage medium include non-volatile data storage carriers, covering Universal Serial Bus flash drives, portable hard drives, and Digital Universal Optical Disk.

[0131] The underlying instruction set operation architecture, bus data transmission arbitration logic, and memory paging scheduling mechanism of the operating system of electronic devices can be implemented by those skilled in the art using existing computer architecture and underlying kernel technology. The underlying hardware level drive and memory fragmentation and allocation methods are well known in the art and will not be described in detail here.

[0132] Specific application examples: Example 1: Specific application of training in the recognition and operation of medical electrocardiogram monitors; Scenario Deployment: This embodiment is applied to the clinical nursing teaching and training room of medical colleges or hospitals. It aims to quickly generate ECG monitor operation training scenarios for students with three-dimensional physical display, spatial sound effects and interactive feedback through an intent-driven mixed reality content automatic generation system.

[0133] Hardware configuration: Configuration terminal: 1 teacher's PC terminal or tablet computer, used to run the front-end visual intent configuration interface.

[0134] Computing cluster: One high-performance edge server (equipped with a tensor processor or graphics processor) deployed with a multimodal large model matrix (3D generation model, TTS model, logical generation engine).

[0135] Rendering and interaction: Multiple mixed reality head-mounted display devices (such as HoloLens2 or similar MR headsets) supporting eye tracking, gesture recognition, and spatial audio playback.

[0136] Workflow and data interaction process: Step 1: Intent input, parsing, and distribution; Intent Input: The teacher inputs the natural language intent on the PC terminal front-end interface: Generate a 3D model of an electrocardiogram (ECG) monitor. When a student touches the device's power button, a voice message indicating that the device has been activated plays, followed by a 2-second delay before updating the interface text to "Please locate the defibrillation electrode patch." When a student correctly clicks on the defibrillation electrode patch using a space ray, 10 points are awarded for the operation.

[0137] Feature Extraction: The intent feature extraction and parsing module converts the above text into intent feature vectors. It extracts features from the prompts generated by the ECG monitor; extracts trigger parameters generated by touch and ray clicks; and extracts metadata features from the logic generated by a 2-second delay and adding 10 minutes.

[0138] Step 2: Multimodal asset collaborative generation and local disk placement Parallel Generation: The AI ​​agent pipeline pushes parameters concurrently. The 3D generated model outputs the original OBJ file of the ECG monitor (approximately 150,000 faces); the text-to-speech model outputs a device-initiated WAV file with spatial reverberation; the logic generation engine encapsulates logic packages containing fractional variables and asynchronous wait instructions.

[0139] Dimension Reduction and Disk Loading: The Local Workspace Asset Synchronization and Hot Replacement module performs mesh dimensionality reduction mapping on OBJ files. The number of polygons was reduced to less than 30,000 (to meet the rendering threshold).

[0140] Record absolute path: The system writes the processed file to the local workspace and records its unique absolute path on the physical disk in the background.

[0141] Step 3: Sentence-based visual logic binding and state machine compilation UI Mapping: The system renders a triplet logical view on the front end. This forms a configuration stream that conforms to natural language sentence structures, such as condition column (detect touch (power), logic column (none), action column (play audio, delay 2 seconds), etc.

[0142] Cross-layer compilation: When a resource is allocated to an action column, the system extracts the previously recorded absolute path pointers and writes them into the component binding mapping table. Based on this, the underlying state machine compilation layer generates a directed graph state machine script consisting of event listeners, logic controllers, and action executors.

[0143] Step 4: On-device unpacking, rendering, and real-time interactive feedback Global Packaging: Package the directed graph state machine, configuration text, and absolute paths into a structured data dictionary compressed package and push it to the MR head-mounted display device.

[0144] Loading and Verification: The runtime engine of the MR headset loads resources directly through the underlying sandbox using an absolute path. When a student puts on the headset and touches the virtual power button, the underlying rendering engine responds instantly, accurately triggering sound effects and scoring actions to complete the interaction loop.

[0145] Experimental verification and effect comparison: To verify the actual effectiveness of this system in the automatic generation of mixed reality content and the execution of underlying logic, multiple comparative experiments were conducted in the aforementioned training environment.

[0146] Control group: Utilized a traditional mainstream commercial game engine (such as Unity3D) for manual development pipeline. This involved manual modeling, writing C# interaction scripts, and using relative paths within the project directory for asset binding.

[0147] Experimental Group: Utilizing the intent-driven automatic mixed reality content generation system of this invention. It employs large-model automatic asset generation, built-in mesh dimensionality reduction optimization, and absolute path penetration binding with cross-layer state machine compilation.

[0148] Experimental data presentation: See attached document Figure 3 Chart description: The horizontal axis represents the test method group (traditional relative path method vs. absolute path addressing in this application), and the vertical axis represents the resource loading addressing error rate (%) under multi-level deeply nested directories.

[0149] Data Interpretation: In the graph, the light gray bars represent the control group, and the dark gray bars represent the experimental group. In 1000 mixed reality scenario thermal overload tests, the control group recorded 142 errors, while the experimental group recorded 0 errors.

[0150] Experimental results: Traditional methods suffer from a 14.2% addressing error rate due to root directory drift during multiple hot reloads and cross-sandbox device migrations. The method proposed in this application achieves a 0% addressing error rate through a cross-layer absolute path binding mechanism, eliminating the resource link breakage problem.

[0151] See attached document Figure 4 Chart description: The horizontal axis represents 5 AI-generated 3D test scenarios with increasing complexity, the vertical axis represents the average rendering frame rate (FPS) on mixed reality devices, and the dashed line represents the 60FPS anti-dizziness smooth baseline.

[0152] Data Interpretation: The light gray solid line marked with square data points represents the unoptimized original generated assets (average polygon count exceeding 250,000), whose frame rates are all below the baseline. The dark gray solid line marked with circular data points represents the optimized assets processed by the experimental group's polygon reduction algorithm, whose frame rates are stable above the baseline.

[0153] In contrast, the unprocessed control group assets caused device rendering stuttering (average of only 34.5 FPS), making normal interaction impossible. The experimental group, however, through pre-processing dimensionality reduction optimization of the local workspace asset synchronization and hot-swap module, increased the average rendering frame rate to 71.2 FPS, perfectly ensuring a low-latency experience.

[0154] Effect Comparison Summary Table ; Conclusion: Experimental results show that the system and method of this invention not only greatly reduce the development threshold of mixed reality content through natural semantic parsing, but also solve the core pain points of AI-generated assets being unable to run on mobile MR devices and being easily lost through surface reduction optimization and absolute path binding mechanism, thereby improving the generation efficiency and operational robustness of MR applications.

Claims

1. An intent-driven method for automatically generating mixed reality content, characterized in that, Includes the following steps: The intent feature extraction and parsing module parses structured intent data and generates a machine-generated parameter set containing trigger parameters, prompt word features, and logical metadata. The multimodal asset collaborative generation module calls the multimodal large model matrix to process the machine-generated parameter set, generating a multimodal asset set containing the original 3D entity file, spatial audio file, and logical package metadata; The local workspace asset synchronization and hot replacement module reduces the original 3D entity file and outputs an optimized 3D entity file. The optimized 3D entity file and the spatial audio file are written to the local workspace directory and their absolute paths are extracted. The sentence-based visual logic binding module constructs a triplet logic chain skeleton based on the trigger parameters, the logic metadata, and the logic package metadata, writes the absolute path into the component binding mapping table, and compiles to generate a directed graph state machine. The mixed reality rendering and real-time preview module encapsulates the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, packages them into a compressed archive file, and pushes it to the mixed reality terminal to perform spatial rendering, thereby realizing the automatic generation of intent-driven mixed reality content.

2. The intention-driven automatic generation method for mixed reality content according to claim 1, characterized in that, The specific steps of the intent feature extraction and parsing module in parsing structured intent data and generating a machine-generated parameter set containing trigger parameters, cue word features, and logical metadata include: The intent feature extraction and parsing module transforms the intent data nodes in the structured intent data into intent feature vectors that include timeline sequence features, target description features, activity type features, and evaluation mechanism features. The named entity recognition algorithm is used to extract entity nouns and attribute modifiers from the target description features and assemble them into the prompt word features. Extract the interaction action identifier from the activity type features to generate the trigger parameters that serve as activation conditions for the underlying state machine interaction events; The timeline sequence features are transformed into time delay parameters in the state machine execution flow, and the spatial distance constraints and interactive feedback indicators in the evaluation mechanism features are transformed into judgment thresholds for conditional branches in the underlying execution code. The time delay parameters and the judgment thresholds are then combined to form the logical metadata. The trigger parameters, the prompt word features, and the logical metadata are combined to generate the machine-generated parameter set.

3. The intention-driven automatic generation method for mixed reality content according to claim 1, characterized in that, Also includes: After writing the optimized 3D entity file and the spatial audio file into the local workspace directory, the local workspace asset synchronization and hot-swap module performs a hot-swap step, specifically including: The local workspace asset synchronization and hot-swap module starts a background listening process to call the underlying operating system's file system application interface to capture file modification events generated in the local workspace directory. When a file modification event triggered by the replacement of an external resource file with the same name is captured, the external resource file with the same name is loaded into the running memory as new multimodal asset data according to the absolute path of the external resource file with the same name obtained. Extract and cache the set of logical components and spatial transformation matrix attached to the original target scene entity within the scene node tree of the mixed reality rendering engine; The new multimodal asset data is used to overwrite the mesh renderer data or audio stream data of the original target scene entity to perform a hot replacement, and the cached set of logical components and the spatial transformation matrix are rebound to the updated scene entity.

4. The intention-driven automatic generation method for mixed reality content according to claim 1, characterized in that, The specific steps of the sentence-based visual logic binding module in constructing a triplet logic chain skeleton and compiling it into a directed graph state machine based on the trigger parameters, the logic metadata, and the logic package metadata include: The sentence-based visual logic binding module constructs a visual view consisting of a condition column panel carrying trigger condition nodes, a logic column panel carrying logic processing nodes, and an action column panel carrying execution action nodes, which are horizontally spliced ​​together. It maps the trigger parameters to the trigger condition nodes, parses the logic metadata into the logic processing nodes, and configures the execution action nodes according to the logic package metadata. When an operation is detected that the optimized 3D entity file or the spatial audio file is assigned to the action column panel or the condition column panel, the sentence-based visual logic binding module penetrates the application layer to directly access the local operating system's file system to obtain the absolute path of the optimized 3D entity file or the spatial audio file, and writes the absolute path into the corresponding execution action node or trigger condition node. The sentence-based visual logic binding module transforms the execution action node into an action executor through the underlying state machine compilation layer, maps the trigger condition node into an event listener component, and transforms the logic processing node into a logic controller, outputting the directed graph state machine composed of the event listener component, the logic controller, and the action executor.

5. The intention-driven automatic generation method for mixed reality content according to claim 1, characterized in that, The specific steps of the mixed reality rendering and real-time preview module in encapsulating the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, packaging them into a compressed archive file, and pushing it to the mixed reality terminal for spatial rendering include: The mixed reality rendering and real-time preview module maps the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into key-value pair format through serialization operations to form the structured data dictionary. It then calls a lossless compression algorithm to convert the structured data dictionary into the compressed archive file and pushes it to the mixed reality terminal through the network communication interface. The runtime engine of the mixed reality rendering and real-time preview module performs an unpacking operation on the received compressed archive file and reconstructs a scene node tree object containing hierarchical relationships and component binding mapping tables by deserializing the structured data dictionary; The underlying file input / output application interface is called according to the absolute path recorded in the component binding mapping table to load the optimized 3D entity file and the spatial audio file into the running memory of the mixed reality device; The optimized 3D entity file and the spatial audio file are injected into the spatial rendering pipeline for spatial transformation and fragment shading calculations, and the directed graph state machine is run concurrently with the concurrent rendering of graphics and audio.

6. The intention-driven automatic generation method for mixed reality content according to claim 2, characterized in that, The intent feature extraction and parsing module uses a parameter parser to parse the intent feature vector to generate the machine-generated parameter set, uses a task scheduler to establish parallel asset generation tasks, and routes them to the multimodal large model matrix through an application programming interface. The multimodal asset collaborative generation module adds a two-channel output command and environmental reverberation parameters to the request header when performing audio generation, thereby generating the spatial audio file carrying three-dimensional spatial positioning attributes.

7. The intention-driven automatic generation method for mixed reality content according to claim 4, characterized in that, The sentence-based visual logic binding module calls and executes a cross-layer automatic binding algorithm at the underlying level to directly map the absolute path to the generated component binding mapping table.

8. The intention-driven automatic generation method for mixed reality content according to claim 1, characterized in that, The mixed reality rendering and real-time preview module uses the global packaging module of the execution resource archive to call the serialization mapping function to convert the structured data dictionary in the runtime memory into a continuous byte stream, and merges it into the compressed archive file according to the absolute path; The mixed reality endpoint utilizes a spatial audio mixing pipeline and a graphics rendering pipeline based on a head-related transfer function to receive and concurrently render the optimized 3D entity file and the spatial audio file loaded based on the absolute path.

9. An intent-driven mixed reality content automatic generation system, characterized in that, The intention-driven automatic generation method for mixed reality content, applied to any one of claims 1-8, comprises: The intent feature extraction and parsing module is used to parse structured intent data and generate a machine-generated parameter set containing trigger parameters, prompt word features, and logical metadata. The multimodal asset collaborative generation module is communicatively connected to the intent feature extraction and parsing module. It is used to call the multimodal large model matrix to process the machine-generated parameter set and generate a multimodal asset set containing the original 3D entity file, spatial audio file and logical package metadata. The local workspace asset synchronization and hot replacement module is communicatively connected to the multimodal asset collaborative generation module. It is used to reduce the dimensionality of the original 3D entity file and output the optimized 3D entity file, write the optimized 3D entity file and the spatial audio file into the local workspace directory, and extract the absolute paths of the optimized 3D entity file and the spatial audio file. The sentence-based visual logic binding module is communicatively connected to the intent feature extraction and parsing module and the local workspace asset synchronization and hot replacement module. It is used to construct a triplet logic chain skeleton based on the trigger parameters, the logic metadata and the logic package metadata, write the absolute path into the component binding mapping table, and compile to generate a directed graph state machine. The mixed reality rendering and real-time preview module is connected to the sentence-based visual logic binding module. It is used to encapsulate the directed graph state machine, the optimized 3D entity file loaded based on the absolute path, and the spatial audio file into a structured data dictionary, package them into a compressed archive file, and push them to the mixed reality terminal for spatial rendering.

10. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the intention-driven automatic generation method for mixed reality content as described in any one of claims 1-8.

Images