A method, system and product of an XR space reconstruction and virtual-real interaction gateway

Abstract

The application discloses a kind of XR space reconstruction and virtual-real interaction gateway method, system and product, obtain voice, text and space perception data form multimodal input data stream, generate candidate intent before natural language input is completed and trigger resource prefetch, cache warm-up, rendering preparation or computing power pre-allocation;According to candidate intent and environmental context, score digital asset matching, select search rendering path or asset missing event path, and support generative asset generation, backfill and version management;Based on depth information, semantic boundary and geometric prior, perform spatial geometry fitting, pixel-level occlusion synthesis and end-side re-projection;When candidate intent involves physical device control or space event linkage presentation, risk score, permission verification, security envelope limiting, execute receipt and audit evidence through semantic security gateway.The application can reduce XR interaction delay, improve virtual-real fitting stability, and form a virtual-real interaction bottom gateway for space computing.

Description

Technical Field

[0001] This invention relates to the fields of extended reality (XR) and spatial computing technology, specifically to methods, systems, and products for XR spatial reconstruction and virtual-real interaction gateways based on multimodal input, predictive intent prediction, edge-cloud collaborative resource scheduling, digital asset retrieval and generation backfilling, spatial geometric fitting and pixel-level occlusion synthesis, edge-focused projection, semantic security gateways, and virtual-real to physical control. Background Technology

[0002] With the development of AI glasses, XR headsets, mobile terminals, and in-vehicle display devices, extended reality (XR) is gradually evolving from displaying preset content to being driven by natural language, multimodal interaction, real-time generation of spatial content, and interaction with physical devices. When users issue voice or text commands in a real space, the system is expected to quickly understand the intent, retrieve or generate corresponding digital content, and maintain stable spatial position, occlusion relationships, and visual consistency in the real environment. When the commands further involve lighting, fans, mechanical structures, in-vehicle equipment, industrial actuators, robots, or unmanned systems, it is also necessary to ensure the safety, traceability, and auditability of the control process.

[0003] The existing technology has the following main shortcomings:

[0004] 1. The response link is biased towards serial, resulting in high end-to-end latency.

[0005] Existing XR systems typically execute semantic understanding, asset retrieval or generation, rendering, encoding, transmission, and display sequentially only after the user's command is completed. This approach causes resource preparation to lag behind semantic input, making it difficult to meet the requirements of XR scenarios for immediate feedback and low-latency rendering. Especially in cloud or edge rendering scenarios, network transmission, encoding / decoding, and rendering queuing further amplify latency.

[0006] 2. The terminal asset pool is limited, lacking a closed loop for asset loss.

[0007] Many XR applications rely on pre-installed 3D models, material packs, effects packs, and interaction scripts on the device itself. Due to limited device storage, computing power, and update capabilities, this approach struggles to support long-tail digital assets and rapidly expand content based on user intent. When the required materials, styles, masking operators, or interaction scripts are unavailable, existing systems typically only return a failure message or offer alternative materials, lacking a complete mechanism to recognize asset absence as a system event and further trigger preview, authorization, generation, backfilling, and version control.

[0008] 3. Insufficient ability to combine spatial fit and occlusion.

[0009] Existing XR solutions often employ two-dimensional overlay, coarse-grained depth testing, or simple occlusion methods. While the virtual content is displayed in the real space, it is prone to errors in occlusion relationships, white edges, glitches, drift, and patchiness in complex environments. Especially when the boundaries of the target physical entity are complex, lighting changes, crowds pass through, or the device turns its head quickly, simple overlay alone is insufficient to achieve stable alignment between the virtual content and the physical entity boundaries.

[0010] 4. Inconsistent timing between edge and cloud, resulting in insufficient stability in dynamic environments.

[0011] When rendering results from the cloud or edge are transmitted back to the terminal, the user's current pose, viewing angle, and environmental occlusion status have often changed. If existing solutions cannot perform pose compensation, reprojection, and local mask refresh based on rendering timestamps, real-time IMU data, real-time pose, or the latest depth boundary, they are prone to producing virtual-real misalignment, image jitter, or edge anomalies in scenarios involving rapid head turns, network jitter, and dynamic occlusion changes.

[0012] 5. Insufficient computing power scheduling and cost control in high-concurrency scenarios.

[0013] In multi-user scenarios such as scenic spots, exhibition halls, parks, and train stations, a large number of users may trigger similar XR requests at close intervals. If the system lacks budgeted management of preheating requests, GPU time slices, video memory quotas, concurrency quotas, and cache quotas, it can easily lead to cloud or edge computing congestion, request queuing, uncontrolled costs, and a decline in service quality.

[0014] 6. The transition from virtual to physical control lacks a security gateway and auditing mechanism.

[0015] When XR systems are further integrated with IoT devices, vehicle-mounted devices, industrial actuators, robots, or unmanned systems, existing solutions mostly focus on protocol issuance or device control itself, lacking risk scoring, permission verification, strong confirmation, security envelope limiting, failure rollback, and audit evidence preservation mechanisms for semantic commands. Voice or text commands are susceptible to misidentification, forgery, replay, or unauthorized triggering. Without tiered security controls and a traceable chain of evidence, physical security risks and difficulties in tracing responsibility can easily arise.

[0016] Therefore, existing technologies urgently need an XR spatial reconstruction and virtual-real interaction gateway solution that can predictively determine intent and preheat resources before the natural language input stream is complete, reduce response latency through edge-cloud collaboration, expand asset supply through digital asset retrieval, asset missing events, generation backfilling, and version governance, improve spatial realism through geometric fitting, dynamic masking, and pixel-level occlusion synthesis, enhance stability in dynamic environments through edge-focused projection and local masking refresh, and achieve secure and auditable physical device linkage through semantic security gateways and virtual-real to physical interaction gateways. Summary of the Invention

[0017] To address the shortcomings of existing technologies, this invention provides a method, system, and product for XR spatial reconstruction and virtual-real interaction gateways, which solves problems such as high response latency, limited asset supply, insufficient spatial fit, weak robustness to dynamic environments, and insufficient security and auditability of physical control in existing XR systems in natural language interaction, cloud rendering, digital asset provisioning, spatial occlusion compositing, edge-cloud timing alignment, and physical device linkage control.

[0018] This invention performs prefix parsing and predictive intent judgment on multimodal input streams, and initiates resource prefetching, cache warm-up, rendering preparation, or computing power pre-allocation associated with candidate intents before the natural language input stream is completed, so that the resource preparation process and the natural language input process overlap at least partially in time; and forms a closed loop from semantic input, resource scheduling, spatial reconstruction, stable presentation to physical device control by using digital asset retrieval, asset missing events, generative engine invocation, asset backfilling, spatial geometry fitting, pixel-level occlusion synthesis, edge-side re-projection, semantic security gateway, and virtual-to-physical interaction gateway.

[0019] To achieve the above objectives, the present invention adopts the following technical solution:

[0020] A method for XR space reconstruction and virtual-physical interaction gateway includes the following steps:

[0021] S1. Acquire a multimodal input data stream, wherein the multimodal input data stream includes at least one of voice input and text input, and at least one of spatially related visual information, pose information, depth information, inertial measurement information, eye tracking information, gesture information, or positioning information;

[0022] S2. Perform sliding window parsing or prefix parsing on the natural language input stream formed by the voice input or text input, and generate a candidate intent set and its confidence distribution by combining the spatially related information;

[0023] S3. Before the natural language input stream is completed, when the candidate intent set meets the preset triggering conditions, the prefetching, cache warm-up, rendering preparation or computing power pre-allocation of cloud resources, edge resources or local resources associated with the candidate intent is triggered.

[0024] S4. Based on the candidate intent set and the environmental context, perform asset matching scoring in the digital asset index library, and select the retrieval rendering path or missing event path according to the scoring threshold.

[0025] S5. In the retrieval and rendering path, the target digital asset is obtained, and geometric fitting, occlusion modeling, mask generation and pixel-level occlusion synthesis are performed in combination with physical environment geometric information, depth information or semantic boundary information to generate spatial reconstruction output results.

[0026] S6. In the missing event path, generate an asset missing event signal and trigger the generative engine to generate digital assets. After generation, the digital assets are backfilled into the asset library and enter the rendering and compositing output.

[0027] S7. The spatial reconstruction output result is transmitted to the terminal in a streaming form, and the terminal performs pose compensation, reprojection or local masking refresh based on the rendering timestamp and real pose to stably present the virtual-real alignment result.

[0028] S8. When the candidate intent involves physical device control or is determined to be a medium-to-high risk intent, the semantic security gateway process is entered to perform risk scoring, permission verification, confirmation control, security envelope limiting and audit recording on the candidate intent. After the security policy is met, the semantic action is mapped into parameterized device control instructions and sent to the physical device associated with the spatial location or geofence. Device acknowledgments are received and a closed-loop feedback is formed.

[0029] Furthermore, in S2, the system performs time-slicing processing on continuous speech streams, text streams, or multimodal input streams to form semantic prefix features, and combines them with visual environment context, spatial target cues, or terminal pose summaries to generate a set of candidate intentions and their confidence distribution.

[0030] Furthermore, the preset triggering conditions in S3 include at least one of the following: the confidence level of the candidate intent exceeds a preset threshold, the confidence level increment of the candidate intent exceeds a preset threshold, the cumulative confidence level of multiple candidate intents exceeds a preset threshold, the matching degree between the candidate intent and the environmental context exceeds a preset threshold, and the candidate intent meets preset rule conditions.

[0031] Furthermore, when the preset triggering conditions are met, the system initiates resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation before the natural language input stream is completed, so that the resource preparation process and the natural language input process overlap at least partially in time.

[0032] Further, in S2, the terminal performs lightweight feature extraction and segmentation on the input stream to form an intent prefix packet and sends it to the edge node or the cloud. The intent prefix packet includes at least one of the following: speech segment features, text prefix features, candidate verb prefixes, candidate object cues, environmental entity summaries, target entity categories, coarse spatial coordinates, key geometric cues, device pose summaries, confidence estimates, or noise estimates. The edge node or the cloud performs multimodal fusion inference based on the intent prefix packet, constructs a candidate intent set and its confidence distribution, and outputs a set of associated asset tags, spatial target cues, or candidate mask types.

[0033] Furthermore, in S3, before triggering resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation, computing power competition arbitration and preheating budget control are performed on the preheating request; the system determines the priority or comprehensive weight of the preheating request based on at least one of geographical priority, service quality level, business level, device capability, historical hit rate, payment status, request cost estimate, or current resource availability, and allocates GPU time slices, video memory quotas, concurrency quotas, or cache quotas according to the priority or comprehensive weight; when resources are insufficient, preheating degradation is performed according to a tiered degradation strategy, the tiered degradation strategy including at least one of the following: degradation from full-link preheating to index prefetching and video memory reservation, degradation from index prefetching and video memory reservation to index prefetching only or low-definition preview preparation, and degradation from low-definition preview preparation to semantic confirmation or placeholder prompt.

[0034] Further, in step S4, the system retrieves candidate digital assets from the digital asset index based on the candidate intent set and the environmental context, and calculates a matching score between the candidate digital assets and the candidate intents. When the matching score is not lower than the scoring threshold, the system enters the retrieval rendering path, loads the target digital asset and its geometric priors, and enters the masking modeling, geometric fitting, pixel-level occlusion synthesis, encoding backhaul, and terminal reprojection compensation process. When the matching score is lower than the scoring threshold, it is determined to be an asset missing event, enters the missing event path, generates an asset missing event signal, and executes at least one of the following: low-cost preview output, outline placeholder output, semantic prompts, authorization requests, generative engine calls, hierarchical generation, generation result backfilling, or version governance. The generation result backfilling includes registering at least one of the following: semantic tags, geometric priors, authorization information, version number, or cost parameters of the generated asset.

[0035] Furthermore, in S5, the system generates or updates a physical environment representation based on depth field, semantic boundary, mesh, point cloud, or physical entity geometric prior, and performs geometric fitting, deformation, cropping, or resampling on the target digital asset according to the physical environment representation, so that the boundary of the target digital asset is aligned with the target physical entity in the spatial coordinate system; the system generates a dynamic mask according to the physical environment representation, and performs pixel-level occlusion determination and edge fusion on the target digital asset and the physical environment background to suppress white edges, glitches, or jitter.

[0036] Furthermore, in S7, after receiving the return frame, layered rendering result or composite layer, the terminal reads the rendering timestamp carried by the return data, obtains the IMU data or pose data at the current time, calculates the pose difference corresponding to the rendering timestamp, and performs two-dimensional reprojection, depth-assisted three-dimensional reprojection or local masking edge refresh on the virtual layer or composite layer to offset the drift caused by network latency, rapid head turning or dynamic occlusion.

[0037] The present invention also provides an XR spatial reconstruction and virtual-real interaction gateway system for performing the above-described method, comprising at least two subsystems from a terminal-side subsystem, an edge-side subsystem, a cloud-side subsystem, a digital asset and governance subsystem, a spatial reconstruction and occlusion synthesis subsystem, a semantic security gateway subsystem, a virtual-to-physical interaction gateway subsystem, and a computing power competition arbitration and budget control subsystem.

[0038] The terminal-side subsystem is used to perform at least one of the following: multimodal acquisition, intent prefix packet generation, local pose prediction, display output, terminal-side re-projection, local masking refresh, or local degradation.

[0039] The edge-side subsystem is used to perform at least one of the following: near-end caching, lightweight prediction, proximity rendering encoding, regional arbitration, or network outage degradation.

[0040] The cloud subsystem is used to perform at least one of the following: predictive semantic prediction, resource preheating and scheduling, asset retrieval, generative engine invocation, masking or fitting calculation, cloud rendering, real-time encoding or streaming distribution.

[0041] The digital asset and governance subsystem is used to support the retrieval and rendering path, missing event path, result backfilling, metadata governance, or authorization information management.

[0042] The spatial reconstruction and occlusion synthesis subsystem is used to generate or update physical environment representations and perform geometric fitting, mask generation, pixel-level occlusion determination or edge fusion.

[0043] The semantic security gateway subsystem is used to perform risk scoring, permission verification, strong authentication, security envelope limiting, fallback control, or audit evidence storage.

[0044] The virtual-to-physical interaction gateway subsystem is used to map semantic actions into physical parameter vectors or device control commands, and send them to the corresponding physical devices based on spatial location, geofence, or device topology.

[0045] The computing power competition arbitration and budget control subsystem is used to queue, prioritize, allocate budgets, limit traffic, downgrade, or reclaim resources for preheating requests.

[0046] Furthermore, the digital asset and governance subsystem includes at least one of a digital asset index library, a matching scoring and routing decision module, and a generative engine invocation and backfilling module. The digital asset index library stores vectorized indexes and metadata of 3D models, materials, special effects operators, masking operators, interactive scripts, or physical control operators. The metadata includes at least one of semantic tags, geometric priors, adaptation rules, authorization information, cost parameters, or version numbers. The matching scoring and routing decision module is used to calculate the matching score between candidate intents and candidate digital assets, and select the retrieval rendering path or missing event path based on the scoring threshold. The generative engine invocation and backfilling module is used to invoke the generative engine to generate digital assets in the missing event path and backfill the generation results to the asset library for reuse in subsequent requests. The spatial reconstruction and occlusion synthesis subsystem includes at least one of an environmental geometry / semantic modeling module, a geometry fitting and deformation module, and a mask generation and edge blending module.

[0047] Furthermore, the semantic security gateway subsystem includes at least one of the following: an intent risk scoring module, a strong confirmation and access control module, a security envelope limiting and fallback module, and an audit and evidence storage module. The intent risk scoring module is used to calculate a risk score based on the action type, amplitude parameter, target device level, scene context, or anomaly detection result, and select a release, weak confirmation, strong confirmation, rejection, or downgrade strategy according to the risk score. The security envelope limiting and fallback module is used to impose upper bound constraints or rate of change constraints on at least one of the amplitude, duration, rate of change, rotation speed, temperature, brightness, duty cycle, voltage, or displacement of the device control command.

[0048] Furthermore, the virtual-to-physical interaction gateway subsystem includes at least one of the following: a semantic dynamics interpreter, a device mapping and geofence association module, a protocol adaptation and delivery module, and a closed-loop feedback module; the semantic dynamics interpreter is used to convert semantic verbs or action descriptions into physical parameter vectors and their timing control curves; the device mapping and geofence association module is used to determine the target device set based on spatial location, geofence, device topology, or priority strategy; and the protocol adaptation and delivery module is used to deliver device control commands and receive device acknowledgments.

[0049] Furthermore, the computing power competition arbitration and budget control subsystem includes at least one of a preheating budget allocation module, a multi-dimensional weight arbitration module, and a hierarchical degradation and rate limiting module.

[0050] The present invention also provides a product for performing the above method, comprising at least one of a computer-readable storage medium, a program, an XR terminal, an edge computing node, a cloud service platform, a digital asset governance platform, or a virtual-to-physical interaction gateway device;

[0051] The computer-readable storage medium and program are used to cause the processor to perform the above methods;

[0052] The XR terminal includes at least one of the following: a sensor assembly, an intent prefix packet generation and transmission unit, a communication unit for receiving cloud or edge rendering output, a display output unit, and a pose compensation / reprojection unit based on rendering timestamps and real-time poses.

[0053] The edge computing node is used to perform at least one of the following: near-end caching, lightweight prediction, near-end rendering encoding, edge arbitration, or network outage degradation.

[0054] The cloud service platform is used to perform at least one of the following: candidate intent prediction, resource preheating, asset retrieval, generative engine invocation, spatial reconstruction calculation, rendering compositing, real-time encoding, or streaming distribution.

[0055] The digital asset governance platform is used to perform at least one of the following: asset indexing, matching scoring, missing event management, authorization information management, result backfilling, or version governance.

[0056] The virtual-to-physical interaction gateway device is used to perform at least one of the following: semantic action parameterization, device mapping, protocol adaptation, control issuance, device receipt reception, or audit recording.

[0057] The beneficial effects of this invention include:

[0058] 1. This invention enables resource preparation and natural language input processes to overlap in time by initiating resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation before the natural language input stream is completed. This reduces the end-to-end latency from semantic input to spatial presentation and improves the immediacy of XR interaction.

[0059] 2. This invention enables the system to prepare candidate resources in advance when the semantics are not yet fully confirmed by using intent prefix packets, multimodal fusion reasoning, candidate intent confidence distribution, and preset triggering conditions, and improves response stability through collaboration between the end side, edge side, and cloud side.

[0060] 3. This invention uses computing power competition arbitration and preheating budget control to budget the allocation of GPU time slices, video memory quotas, concurrency quotas and cache quotas in scenarios with multiple users in the same domain or with limited resources, and maintains system availability and cost controllability through a tiered degradation strategy.

[0061] 4. This invention engineers asset missingness into a processable system event through a digital asset index library, matching scoring, retrieval rendering path, and missing event path. It also expands the supply of digital assets and enables subsequent reuse through low-cost preview, authorization requests, generative engine calls, hierarchical generation, backfilling, and version management.

[0062] 5. This invention constructs a physical environment representation through depth field, semantic boundary, mesh, point cloud or geometric prior, and performs geometric fitting, deformation, cropping, dynamic masking, pixel-level occlusion judgment and edge blending to keep the virtual content and physical entity boundary stably aligned, reducing white edges, glitches, jitter and patchiness.

[0063] 6. This invention reduces the impact of cloud or edge backhaul latency, rapid head turning, network jitter, and dynamic occlusion changes on the virtual-real alignment effect by rendering timestamps, real-time pose, pose difference calculation, 2D reprojection, depth-assisted 3D reprojection, and local masking edge refresh.

[0064] 7. This invention uses a semantic security gateway to perform risk scoring, permission verification, strong confirmation, security envelope limiting, fallback control, and audit evidence storage for intentions involving physical device control or medium-to-high risk, thereby reducing the security risks caused by false triggering, malicious triggering, or high-risk physical control.

[0065] 8. This invention uses a virtual-to-physical interaction gateway to transform semantic verbs or action descriptions into physical parameter vectors or device control commands, and distributes them to the corresponding physical devices based on spatial location, geofencing, device topology, or priority policies, thereby achieving secure linkage and feedback loop between XR spatial content and IoT devices, vehicle-mounted devices, industrial actuators, robots, or unmanned systems.

[0066] 9. This invention enables the system to cover multiple product forms such as terminals, edges, clouds, digital assets and governance, spatial reconstruction and occlusion synthesis, semantic security gateway, virtual-to-physical interaction gateway, and computing power competition arbitration and budget control through the synergy of terminal-side subsystem, edge-side subsystem, cloud, asset governance, spatial reconstruction, security audit, and physical control, thereby improving platform deployment capabilities and engineering scalability. Attached Figure Description

[0067] Figure 1 This is a topology diagram of the overall architecture of the XR space reconstruction and virtual-real interaction gateway system of the present invention;

[0068] Figure 2 This is an overlay diagram of the predictive intent prediction and resource preheating time series of the present invention;

[0069] Figure 3 This is the state machine diagram for the dual-track resource routing and asset missing event of the present invention;

[0070] Figure 4 This is a schematic diagram illustrating the spatial geometric fitting and pixel-level occlusion synthesis principle of the present invention.

[0071] Figure 5 This is a flowchart of the terminal-side attitude compensation, reprojection, and local masking refresh process of the present invention.

[0072] Figure 6 This is the state machine diagram of the semantic security gateway of the present invention;

[0073] Figure 7 This is a mapping diagram of the virtual-to-physical interaction gateway and geofencing device of the present invention;

[0074] Figure 8 This is a schematic diagram of the data structure and message flow of the present invention. Detailed Implementation

[0075] The embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the described embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments can be combined, substituted, or adjusted with each other.

[0076] Please see Figures 1-8 This invention provides a method, system, and product for XR space reconstruction and virtual-real interaction gateway, which can be applied to AI glasses, XR headsets, mobile terminals, in-vehicle display devices, in-vehicle HUDs, spatial computing terminals, edge computing nodes, cloud service platforms, digital asset governance platforms, as well as IoT devices, in-vehicle devices, industrial actuators, robots, or unmanned systems associated with physical spaces.

[0077] The XR described in this invention includes Augmented Reality (AR), Mixed Reality (MR), Virtual Reality (VR), and their equivalent display forms. The multimodal input described in this invention may include at least one of the following: voice input, text input, image input, video input, depth information, inertial measurement information, pose information, eye-tracking information, gesture information, positioning information, or environmental perception information. The digital assets described in this invention may include 3D models, materials, textures, animations, special effects operators, masking operators, interactive scripts, spatial events, physical control operators, or combinations thereof.

[0078] This invention generates a set of candidate intents and their confidence distributions before the user completes input by performing sliding window parsing or prefix parsing on the natural language input stream. It also initiates resource prefetching, cache warm-up, rendering preparation, or computing power pre-allocation when preset trigger conditions are met. Furthermore, it selects and retrieves rendering paths or missing event paths through digital asset matching and scoring. It achieves spatial alignment between virtual content and the physical environment through spatial geometry fitting, dynamic masking, and pixel-level occlusion synthesis. It reduces network latency and drift caused by rapid head turns and dynamic occlusion changes through edge-side posture compensation, reprojection, or local masking refresh. When candidate intents involve physical device control or medium-to-high-risk outputs, it performs risk scoring, permission verification, strong confirmation, security envelope limiting, device control issuance, device receipt reception, and audit logging through a semantic security gateway and a virtual-to-physical interaction gateway.

[0079] In this invention, the XR includes Augmented Reality (AR), Mixed Reality (MR), Virtual Reality (VR), and their equivalent display forms; the multimodal input data stream includes at least one of voice input, text input, image input, video input, depth information, inertial measurement information, pose information, eye tracking information, gesture information, positioning information, or environmental perception information; the digital assets include 3D models, materials, textures, animations, special effects operators, masking operators, interaction scripts, spatial events, physical control operators, or combinations thereof; the asset missing event refers to a system event generated when the digital assets, masking operators, interaction scripts, physical control operators, or adaptation rules required by the candidate intent are absent, unavailable, or insufficiently matched; the virtual-to-physical interaction gateway refers to a functional unit that converts semantic actions or XR spatial events into physical parameter vectors or device control commands and sends them to the corresponding physical devices.

[0080] This invention is not limited to the synthesis of single XR images, but rather forms a virtual-real interaction gateway platform for spatial computing scenarios through predictive semantics, digital asset governance, edge-cloud resource scheduling, spatial reconstruction, security auditing, and physical device control closed loop.

[0081] 1. Overall Implementation Architecture

[0082] like Figure 1 As shown, the XR spatial reconstruction and virtual-real interaction gateway of the present invention can adopt an end-edge-cloud collaborative architecture, or it can adopt an end-cloud collaborative, end-edge collaborative, local independent execution, or cloud centralized execution architecture according to the actual deployment conditions. The architecture includes a terminal-side execution environment, a resource processing-side execution environment, a digital asset and governance environment, a spatial reconstruction and occlusion synthesis environment, a security control environment, and a physical interaction execution environment.

[0083] The terminal-side execution environment is used to collect user input and spatial awareness data, and is responsible for intent prefix packet generation, display output, terminal-side posture compensation, reprojection, local masking refresh, or local degradation processing. The terminal-side execution environment can be deployed in AI glasses, XR headsets, mobile terminals, in-vehicle display devices, in-vehicle HUDs, or spatial computing terminals. While acquiring natural language input, the terminal-side execution environment can also acquire visual information, depth information, IMU information, pose information, eye-tracking information, gesture information, or positioning information, enabling subsequent candidate intent judgment to combine the user's spatial location, the user's viewing direction, and the target physical object.

[0084] The resource processing side execution environment is used to perform candidate intent prediction, resource preheating, computing power arbitration, asset retrieval, generative engine invocation, rendering compositing, encoding transmission, or streaming distribution. This resource processing side execution environment can be deployed on edge computing nodes, cloud servers, regional rendering nodes, private servers, or local high-performance computing devices. After receiving the intent prefix packet sent by the terminal, the resource processing side execution environment can prepare candidate resources in advance based on the candidate intent set and its confidence distribution before the natural language input is completed, ensuring that the resource preparation process at least partially overlaps with the input process in time.

[0085] The digital asset and governance environment is used to manage the digital asset index, asset metadata, matching and scoring rules, asset missing events, generation result backfilling, authorization information, cost parameters, and version information. The system performs matching and scoring in the digital asset index based on candidate intents and environmental context. When the score meets a threshold condition, it enters the retrieval and rendering path; when the score does not meet the threshold condition or the required asset does not exist, it enters the missing event path. In this way, the system can transform "asset non-existence" into a processable system event, further triggering low-cost previews, semantic hints, authorization requests, generative engine calls, hierarchical generation, backfilling, and version governance.

[0086] The spatial reconstruction and occlusion synthesis environment is used to generate a physical environment representation based on depth fields, semantic boundaries, meshes, point clouds, or physical entity geometric priors. The target digital assets are then geometrically fitted, deformed, cropped, resampled, dynamically masked, pixel-level occlusion determined, and edge-blended according to this physical environment representation. Through this environment, virtual content is not merely superimposed on the terminal screen, but is synthesized based on the geometric boundaries, depth relationships, and occlusion relationships of real space, thereby reducing white borders, artifacts, drift, jitter, or patchiness.

[0087] The security control environment is used to perform risk scoring, permission verification, strong confirmation, security envelope limiting, rollback control, and audit evidence storage for candidate intentions involving physical device control, medium-to-high risk output, unauthorized operation, or abnormal triggering. The risk scoring can be determined by combining action type, action amplitude, target device level, scene context, user identity, device status, environmental status, or anomaly detection results. The security envelope limiting can be used to restrict the amplitude, duration, rate of change, rotational speed, temperature, brightness, duty cycle, voltage, displacement, or range of action of device control commands.

[0088] The physical interaction execution environment, after security policies are approved, transforms semantic actions, spatial events, or interaction intentions into physical parameter vectors or device control commands, and distributes them to physical devices associated with spatial locations, geofences, or device topologies. These physical devices may include IoT devices, lighting equipment, fan equipment, mechanical structures, vehicle-mounted equipment, industrial actuators, robots, or unmanned systems. After execution, the device returns a device acknowledgment, and the system writes the control commands, execution results, confirmation records, risk scores, security envelope parameters, and timestamps into the audit log, thus forming a closed loop from semantic intent to physical execution.

[0089] In one implementation, a user issues voice or text commands via an XR terminal. The terminal-side execution environment receives the input and simultaneously acquires the current pose, IMU data, depth data, and environmental image, generating an intent prefix packet which is sent to the resource processing-side execution environment. The resource processing-side execution environment generates a candidate intent set and its confidence distribution based on the intent prefix packet, and pre-initiates resource prefetching, cache warm-up, rendering preparation, or computing power pre-allocation when preset trigger conditions are met. Subsequently, the system performs asset matching scoring based on the candidate intents and environmental context, selecting a retrieval rendering path or a missing event path. After rendering and compositing are completed, the system transmits the return frames, layered rendering results, or compositing laminar flow to the terminal. The terminal performs pose compensation, reprojection, or local masking refresh based on the rendering timestamp and current pose to achieve virtual-real aligned display.

[0090] In another implementation, the candidate intent involves physical device control. The system first determines the risk level of the candidate intent through a semantic security gateway and executes a pass, weak confirmation, strong confirmation, rejection, or downgrade strategy based on the risk level. After security verification, the virtual-to-physical interaction gateway converts the semantic action into parameterized control commands that the device can recognize, and selects a set of target devices according to spatial location, geofencing, device topology, or priority strategies. After the target device executes the command, it returns a receipt. The system feeds back the device receipt to the terminal-side execution environment, the resource processing-side execution environment, or the security control environment, and generates an audit log.

[0091] In the overall implementation architecture described above, the terminal-side execution environment, resource processing-side execution environment, digital asset and governance environment, spatial reconstruction and occlusion synthesis environment, security control environment, and physical interaction execution environment can be deployed in a distributed manner or integrated into a unified platform; they can be deployed in public clouds, private clouds, dedicated network edge nodes, vehicle-mounted computing devices, local servers, or terminal devices. Different deployment methods do not affect the technical effects of this invention in achieving XR spatial reconstruction and virtual-real interaction through predictive intent prediction, resource preheating, dual-track resource routing, spatial reconstruction, terminal-side compensation, security gateways, and physical control closed loops.

[0092] The aforementioned execution environments can be deployed on terminals, edge computing, cloud computing, private network servers, in-vehicle computing devices, local high-performance computing devices, or combinations thereof. Regardless of whether direct inference using edge models, collaborative inference using edge models, centralized inference using cloud models, or hybrid inference using edge-cloud models, as long as resource preparation, spatial reconstruction, or virtual-real interaction control is triggered based on candidate intents with incomplete input, they can all be included in the implementation of this invention.

[0093] 2. System Composition

[0094] In one embodiment, the XR spatial reconstruction and virtual-real interaction gateway system of the present invention includes a terminal-side subsystem, an edge-side subsystem, a cloud-side subsystem, a digital asset and governance subsystem, a spatial reconstruction and occlusion synthesis subsystem, a semantic security gateway subsystem, a virtual-to-physical interaction gateway subsystem, and a computing power competition arbitration and budget control subsystem. These subsystems can be deployed separately, or integrated and deployed on the same terminal, the same edge node, the same cloud platform, or the same dedicated computing device, depending on the application scenario.

[0095] 2.1 Terminal-side subsystem

[0096] The terminal-side subsystem is used to acquire multimodal input data streams, generate intent prefix packets, receive return frames, layered rendering results or composite layers, and perform display output, terminal pose compensation, reprojection, local masking refresh or local degradation processing.

[0097] The terminal-side subsystem may include at least one of the following: a multimodal acquisition unit, an intent prefix packet generation unit, a communication unit, a display output unit, a posture compensation unit, a reprojection unit, a local masking refresh unit, and a local degradation unit.

[0098] The multimodal acquisition unit is used to acquire voice input, text input, image input, video input, depth information, inertial measurement information, pose information, eye tracking information, gesture information, positioning information, or environmental perception information. The intent prefix packet generation unit is used to encapsulate speech segments, text prefixes, candidate object cues, environmental entity summaries, spatial target cues, device pose summaries, or confidence estimates into intent prefix packets before the natural language input stream is complete, and then send them to the edge subsystem or cloud subsystem.

[0099] The display output unit displays the spatial reconstruction output results. The pose compensation unit and reprojection unit calculate the pose difference based on the rendering timestamp and current pose, and perform 2D reprojection, depth-assisted 3D reprojection, or display position correction. The local mask refresh unit refreshes mask edges, occlusion relationships, or local compositing areas based on the latest depth boundaries, semantic boundaries, or local environmental changes. The local degradation unit performs low-resolution preview, outline placeholders, static asset replacement, interactive pause, or safe rollback when there are network anomalies, insufficient resources, or security policy failures.

[0100] 2.2 Edge Subsystem

[0101] The edge subsystem is used to perform near-end caching, lightweight prediction, proximity rendering, regional arbitration, low-latency code distribution, or network outage degradation processing at locations close to the terminal or close to the scene.

[0102] The edge subsystem may include at least one of the following: a near-end caching unit, a lightweight prediction unit, an edge rendering and encoding unit, a region arbitration unit, a security policy unit, and a network disconnection degradation unit.

[0103] The near-end caching unit is used to cache high-frequency digital assets, commonly used regional masking operators, spatial geometric priors, low-resolution preview results, and regional device topology or geofencing information. The lightweight prediction unit is used to perform candidate intent filtering, candidate asset label generation, or candidate spatial target recognition based on intent prefix packets. The edge rendering and encoding unit is used to perform near-end rendering, encoding, and postback on the hit digital assets, low-resolution preview assets, or local composite results.

[0104] The regional arbitration unit is used to queue, prioritize, allocate budgets, or limit the rate of preheating requests, rendering requests, generation requests, or physical control requests initiated by multiple terminals within the same region. The network outage degradation unit is used to maintain basic XR presentation and security control when cloud connectivity is abnormal or external networks are unavailable, utilizing edge-side cached assets, preset scripts, low-definition preview results, or local security policies.

[0105] 2.3 Cloud Subsystem

[0106] The cloud subsystem is used to perform predictive semantic prediction, resource preheating and scheduling, asset retrieval, generative engine invocation, spatial reconstruction calculation, cloud rendering, real-time encoding, and streaming distribution.

[0107] The cloud subsystem may include at least one of the following: a semantic prediction unit, a resource preheating and scheduling unit, an asset retrieval unit, a generative engine invocation unit, a spatial reconstruction calculation unit, a cloud rendering unit, a real-time encoding unit, and a streaming distribution unit.

[0108] The semantic prediction unit receives the intent prefix packet and, in conjunction with the environmental context, spatial target cues, scene rules, or the user's current pose, generates a set of candidate intents and their confidence distribution. The resource preheating and scheduling unit is used to pre-start the pre-fetching, cache preheating, rendering preparation, or computing power pre-allocation of cloud resources, edge resources, or local resources when the set of candidate intents meets preset triggering conditions.

[0109] The asset retrieval unit retrieves candidate digital assets from the digital asset index based on candidate intent, environmental context, and asset tags, and outputs a matching score. The generative engine invocation unit generates 3D models, materials, textures, animations, effects, masking operators, interactive scripts, spatial events, or physical control operators when an asset is not matched, the matching score is insufficient, or an asset missing event is triggered. The spatial reconstruction calculation unit and cloud rendering unit perform geometric fitting, mask generation, pixel-level occlusion compositing, and rendering output based on depth fields, semantic boundaries, meshes, point clouds, or geometric priors. The real-time encoding unit and streaming distribution unit encode, timestamp, and stream the rendering results.

[0110] 2.4 Digital Assets and Governance Subsystem

[0111] The Digital Assets and Governance Subsystem is used to manage the digital asset index, asset metadata, matching scoring rules, asset missing events, generation result backfilling, authorization information, cost parameters, and version information.

[0112] The digital asset and governance subsystem may include at least one of the following: digital asset index repository, metadata management unit, matching scoring and routing decision unit, asset missing event management unit, generation result backfilling unit, authorization and cost management unit, and version governance unit.

[0113] The digital asset index is used to store vectorized indexes and associated data of 3D models, materials, textures, animations, effects operators, masking operators, interaction scripts, spatial events, physical control operators, or combinations thereof. The metadata management unit is used to manage the semantic tags, geometric priors, adaptation rules, occlusion rules, interaction rules, licensing information, cost parameters, version numbers, applicable scenarios, or quality levels of digital assets.

[0114] The matching score and routing decision unit calculates the matching score between candidate intents and candidate digital assets, and selects either the retrieval rendering path or the missing event path based on the score threshold. When the matching score is not lower than the score threshold, the system enters the retrieval rendering path; when the matching score is lower than the score threshold, or the required asset does not exist, is unavailable, has insufficient authorization, or is of insufficient quality, the system enters the missing event path.

[0115] The Asset Missing Event Management Unit generates asset missing event signals and records the missing type, candidate intent, environmental context, target entity category, required asset type, authorization status, or generation requirement. The Generation Result Backfilling Unit registers the digital assets output by the generative engine back into the digital asset index and writes semantic tags, geometric priors, adaptation rules, authorization information, cost parameters, version numbers, or quality identifiers. The Version Governance Unit records, rolls back, replaces, performs quality audits, or handles conflicts for generation results, masking operators, interaction scripts, physical control operators, or adaptation rules.

[0116] The digital asset index library is not limited to an explicitly stored 3D model library, but may also include a parametric asset template library, a set of callable operators for generative models, asset representations that can be generated in real time by generative models, spatial event scripts, interaction scripts, masking operators, or physical control operators.

[0117] 2.5 Spatial Reconstruction and Occlusion Synthesis Subsystem

[0118] The spatial reconstruction and occlusion synthesis subsystem is used to generate or update physical environment representations, and to perform geometric fitting, deformation, cropping, resampling, dynamic mask generation, pixel-level occlusion determination and edge blending of target digital assets based on the physical environment representations.

[0119] The spatial reconstruction and occlusion synthesis subsystem may include at least one of the following: an environmental geometry modeling unit, a semantic boundary recognition unit, a geometry fitting and deformation unit, a mask generation unit, a pixel-level occlusion determination unit, an edge blending unit, and a dynamic update unit.

[0120] The environment geometry modeling unit is used to generate physical environment representations based on depth fields, meshes, point clouds, planar detection results, physical entity geometric priors, or scene calibration data. The semantic boundary recognition unit is used to identify the contours, boundaries, alternative regions, non-occluded regions, foreground regions, background regions, or interactive regions of target physical entities.

[0121] The geometric fitting and deformation unit is used to perform spatial transformations, boundary alignment, deformation, cropping, or resampling on the target digital asset according to the physical environment representation, so that the target digital asset matches the target physical entity or target physical region in the spatial coordinate system. The mask generation unit is used to generate binary masks, depth masks, semantic masks, edge masks, or multi-layer masks based on depth information, semantic boundaries, geometric priors of the target asset, or dynamic environmental changes.

[0122] The pixel-level occlusion determination unit is used to determine the pixel-level occlusion relationship between the virtual content and the physical environment background based on the virtual content depth, physical environment depth, masking data, or semantic boundaries. The edge blending unit is used to perform feathering, anti-aliasing, depth transitions, color transitions, or brightness transitions at the boundary between the virtual content and the physical environment to reduce white edges, artifacts, drift, jitter, or patchiness. The dynamic update unit is used to update the physical environment representation, masking data, or local compositing areas in response to changes in user pose, physical entity movement, environmental occlusion, lighting, or network latency.

[0123] The mask generation and pixel-level occlusion synthesis are not limited to traditional depth masking methods, but can also be achieved through semantic segmentation models, neural rendering models, optical flow compensation models, multi-layer synthesis models, or combinations thereof.

[0124] 2.6 Semantic Security Gateway Subsystem

[0125] The semantic security gateway subsystem is used to perform risk scoring, permission verification, strong confirmation, security envelope limiting, rollback control, and audit evidence storage on candidate intents when the candidate intents involve physical device control, medium-to-high risk output, unauthorized operation, or abnormal triggering.

[0126] The semantic security gateway subsystem may include at least one of the following: an intent risk scoring unit, an authorization verification unit, a strong confirmation unit, a security envelope limiting unit, a fallback control unit, and an audit and evidence storage unit.

[0127] The intent risk scoring unit calculates a risk score for candidate intents based on action type, action amplitude, target device level, scene context, user identity, device status, environmental status, anomaly detection results, or historical behavior records. Based on the risk score, it selects a grant, weak confirmation, strong confirmation, denial, or downgrade strategy. The permission verification unit verifies user permissions, device permissions, spatial permissions, time permissions, scene permissions, or operational permissions.

[0128] The strong confirmation unit is used to trigger voiceprint confirmation, iris confirmation, device binding confirmation, administrator review, physical button confirmation, time window secondary confirmation, or other confirmation methods in medium-to-high risk intent or high-risk device control scenarios. The safety envelope limiting unit is used to impose upper, lower, or rate-of-change constraints on at least one of the following: amplitude, duration, rate of change, rotation speed, temperature, brightness, duty cycle, voltage, displacement, or range of action of device control commands or high-risk outputs. The rollback control unit is used to execute rejection, stop, downgrade, restore default state, switch security mode, or output prompts when the risk score is too high, permission verification fails, strong confirmation fails, device malfunctions, environmental malfunctions, or execution exceeds limits. The audit and evidence storage unit is used to record candidate intents, risk scores, permission verification results, confirmation results, safety envelope parameters, control commands, device receipts, execution results, and timestamps.

[0129] 2.7 Virtual-to-Physical Interaction Gateway Subsystem

[0130] The virtual-to-physical interaction gateway subsystem is used to convert semantic actions, XR spatial events, or interaction intentions that have been verified by security policies into physical parameter vectors or device control commands, and then send them to physical devices associated with spatial location, geofence, or device topology.

[0131] The virtual-to-physical interaction gateway subsystem may include at least one of the following: a semantic dynamics interpreter, a device mapping and geofencing association unit, a protocol adaptation and distribution unit, a device status feedback unit, and a closed-loop feedback unit.

[0132] A semantic dynamics interpreter is used to convert semantic verbs, action descriptions, spatial events, or interaction intentions into physical parameter vectors and their timing control curves. The physical parameter vectors may include at least one of the following: target device identifier, action type, amplitude, direction, duration, rate of change, start time, stop time, range of action, or priority.

[0133] The device mapping and geofencing association unit is used to determine the target device set based on spatial location, geofencing, device topology, device status, scene rules, or priority policies. The protocol adaptation and delivery unit is used to convert physical parameter vectors into protocol commands recognizable by the target devices and send them to the target devices via wired networks, wireless networks, local area networks, private networks, cellular networks, vehicular networks, or device buses. The device status feedback unit is used to receive execution status, fault status, limit status, temperature status, current status, duty cycle status, location status, or other device feedback returned by the target devices. The closed-loop feedback unit is used to feed back device feedback, execution status, or abnormal status to the terminal-side subsystem, cloud subsystem, semantic security gateway subsystem, or audit and evidence storage unit.

[0134] The target device is not limited to IoT devices, but may also include lighting equipment, fan equipment, audio equipment, mechanical structures, vehicle-mounted equipment, industrial actuators, robots, unmanned vehicles, drones, or other controllable physical devices associated with spatial location, geofence, or device topology; in some embodiments, the execution result of the target device may also be combined with spatial event presentation, digital twin object state update, or virtual content state update bound to physical space.

[0135] 2.8 Computing Power Competition Arbitration and Budget Control Subsystem

[0136] The computing power contention arbitration and budget control subsystem is used to queue, prioritize, allocate budgets, rate limit, degrade, and reclaim resources in scenarios involving multiple users, multiple requests, multiple candidate intents, or multiple resource pools, for requests such as resource prefetching, cache preheating, rendering preparation, generative engine invocation, encoding transmission, or physical control.

[0137] The computing power competition arbitration and budget control subsystem may include at least one of the following: a request queue management unit, a multi-dimensional weight arbitration unit, a preheating budget allocation unit, a hierarchical degradation and rate limiting unit, a resource recycling unit, and a statistical update unit.

[0138] The request queue management unit receives preheating requests, rendering requests, generation requests, encoding requests, asset retrieval requests, or device control requests from the terminal-side subsystem, edge-side subsystem, or cloud subsystem, and establishes a request queue. The multi-dimensional weight arbitration unit determines the priority or comprehensive weight of requests based on at least one of the following: geographical priority, service quality level, business level, device capacity, historical hit rate, payment status, request cost estimate, current resource availability, scene congestion level, or security level.

[0139] The preheating budget allocation unit is used to allocate GPU time slices, memory quotas, concurrency quotas, cache quotas, generative model call quotas, encoding bandwidth, or other computing resources based on request priority or comprehensive weight. The tiered degradation and rate limiting unit is used to execute tiered degradation strategies when resources are insufficient, requests are congested, costs exceed limits, or security levels are restricted. These tiered degradation strategies include at least one of the following: degradation from full-link preheating to index prefetching and memory reservation; degradation from index prefetching and memory reservation to index prefetching only or low-resolution preview preparation; and degradation from low-resolution preview preparation to semantic confirmation or placeholder hints.

[0140] The resource reclamation unit is used to release cache, video memory, GPU time slices, concurrency limits, or model call limits after candidate intents are not hit, user requests are cancelled, warm-up times out, rendering is complete, generation is complete, session ends, or degradation is complete. The statistics update unit is used to record prediction hit rate, warm-up hit rate, resource usage, average latency, failure rate, degradation count, generation cost, cache hit rate, user confirmation results, or security interception results, and uses the statistical results for subsequent resource scheduling, score updates, or strategy optimization.

[0141] 3. Main Method Flow

[0142] In one embodiment, the XR space reconstruction and virtual-real interaction gateway method of the present invention includes the following steps.

[0143] S1. Acquisition of multimodal input data stream

[0144] The terminal-side subsystem acquires user input and spatial awareness data to form a multimodal input data stream. The multimodal input data stream includes at least one of voice input and text input, as well as at least one of visual information, pose information, depth information, inertial measurement information, eye-tracking information, gesture information, positioning information, or environmental awareness information.

[0145] The terminal-side subsystem can perform timestamp marking, noise filtering, data slicing, preliminary coordinate alignment, or lightweight feature extraction on the multimodal input data stream, so that natural language input and spatial awareness information enter the same processing link, providing input for subsequent candidate intent generation, resource preheating, asset matching, spatial reconstruction, and terminal-side compensation.

[0146] S2, Candidate Intent Generation

[0147] The system performs sliding window parsing or prefix parsing on the natural language input stream formed by voice input or text input, and combines visual environment context, spatial target cues, device pose summary, depth boundary information, semantic boundary information or scene rule information to generate a set of candidate intents and their confidence distribution.

[0148] The sliding window parsing refers to the system continuously segmenting the natural language input stream according to a preset time length, speech segment length, text character length, or semantic unit length, and performing semantic judgment on the current segment or adjacent segment combinations. The prefix parsing refers to the system performing phased semantic inference on received speech segments, text prefixes, candidate verbs, candidate objects, spatial pointing information, or contextual information before the user has completed complete speech or text input.

[0149] Through this step, the system can obtain one or more possible candidate intentions before a complete instruction is formed, and associate the candidate intentions with spatial targets, digital asset types, rendering methods, occlusion types, interaction scripts, or physical control types.

[0150] S3, resource prefetching, cache preheating, and computing power pre-allocation

[0151] Before the natural language input stream is completed, when the candidate intent set meets the preset triggering conditions, the system triggers resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation of cloud resources, edge resources, or local resources associated with the candidate intent.

[0152] The preset triggering conditions may include at least one of the following: the confidence level of the candidate intent exceeds a preset threshold; the confidence level increment of the candidate intent exceeds a preset threshold; the cumulative confidence level of multiple candidate intents exceeds a preset threshold; the matching degree between the candidate intent and the environmental context exceeds a preset threshold; and the candidate intent satisfies a preset rule condition.

[0153] The resource prefetching can include digital asset index prefetching, candidate asset metadata prefetching, geometric prior prefetching, masking operator prefetching, interactive script prefetching, spatial event prefetching, or device topology information prefetching. The cache warm-up can include loading candidate digital assets, low-resolution preview assets, materials, textures, animations, effects operators, masking operators, spatial events, or physical control operators into the terminal cache, edge cache, cloud cache, or video memory cache. The rendering preparation can include rendering context creation, rendering queue establishment, candidate compositing layer preparation, masking computation task preparation, low-resolution preview result preparation, or encoding pipeline preparation. The computing power pre-allocation can include GPU time slice reservation, video memory quota reservation, concurrency quota reservation, generative model call quota reservation, encoding bandwidth reservation, or edge computing resource reservation.

[0154] This step ensures that the resource preparation process overlaps with the natural language input process at least partially in time, thereby reducing the waiting time caused by the user completing the instruction and then initiating semantic understanding, asset retrieval, generation invocation, rendering preparation, or physical control verification.

[0155] S4, Asset Matching Scoring and Dual-Track Resource Routing

[0156] Based on the candidate intent set and environmental context, the system performs asset matching and scoring in the digital asset index, and selects the retrieval rendering path or missing event path according to the scoring threshold.

[0157] During the asset matching process, the system can retrieve candidate digital assets from the digital asset index based on the semantic tags of the candidate intent, the category of the target entity, the spatial target location, the user's perspective, the scene type, the required asset type, the interaction method, the occlusion requirement, the authorization status, the cost parameters, or the historical hit records, and calculate the matching score between the candidate digital assets and the candidate intent.

[0158] When the matching score is not lower than the score threshold, the system enters the retrieval rendering path, loads at least one of the target digital assets, geometric priors, materials, textures, animations, special effects operators, masking operators, interactive scripts, spatial events, or physical control operators, and enters the spatial reconstruction and rendering compositing process.

[0159] When the matching score is lower than the scoring threshold, or when the digital assets, masking operators, interactive scripts, physical control operators, or adaptation rules required by the candidate intent are missing, unavailable, insufficiently authorized, or of insufficient quality, the system enters the missing event path, generates an asset missing event signal, and records the missing type, candidate intent, environmental context, target entity category, required asset type, authorization status, or generation requirements.

[0160] S5, Spatial Geometric Fitting and Pixel-Level Occlusion Synthesis

[0161] In the retrieval and rendering path, the system acquires the target digital asset and combines it with physical environment geometric information, depth information or semantic boundary information to perform geometric fitting, occlusion modeling, mask generation and pixel-level occlusion synthesis to generate spatial reconstruction output results.

[0162] Specifically, the system can generate a physical environment representation based on depth field, semantic boundary, mesh, point cloud, plane detection results, physical entity geometric prior or scene calibration data; then, the target digital asset is spatially transformed, boundary aligned, deformed, clipped or resampled according to the physical environment representation, so that the target digital asset matches the target physical entity or target physical region in the spatial coordinate system.

[0163] During the occlusion compositing process, the system performs pixel-level occlusion relationship determination between virtual content and physical environment background based on virtual content depth, physical environment depth, masking data, semantic boundaries or geometric priors, and performs edge blending, anti-aliasing, depth transition, color transition, brightness transition or local transparency adjustment at the boundary to reduce white edges, glitches, drift, jitter or patchiness.

[0164] S6. Generation, backfilling, and output of missing event paths

[0165] In the missing event path, the system triggers at least one of the following based on the asset missing event signal: low-cost preview, outline placeholder, semantic hint, authorization request, generative engine call, hierarchical generation, generation result backfilling, or version governance.

[0166] In one implementation, when the system determines that the target asset does not exist or the match is insufficient, it first outputs a low-cost preview, outline placeholder or semantic prompt to the terminal to provide the user with immediate feedback; at the same time, based on the candidate intent, environmental context, target entity category, spatial boundary, style constraint, interaction requirements, authorization status or cost budget, it calls the generative engine to generate the corresponding digital asset.

[0167] The digital assets output by the generative engine can include 3D models, materials, textures, animations, special effects, masking operators, interactive scripts, spatial events, or physical control operators. The system performs quality checks, format conversions, geometric prior registration, semantic tag registration, licensing information registration, cost parameter registration, or version number registration on the generated results. The generated results are then populated back into the digital asset index for continued use in the current request or reuse in subsequent requests. Once the generated results meet rendering or interactive requirements, the system sends them to the spatial reconstruction and occlusion compositing workflow, generating the spatial reconstruction output.

[0168] S7, streaming, attitude compensation and end-weighted projection

[0169] The system transmits the spatial reconstruction output to the terminal in a streaming format. The spatial reconstruction output may include at least one of the following: return frames, layered rendering results, compositing layers, masking data, depth auxiliary information, reprojection parameters, rendering timestamps, or local refresh parameters.

[0170] After receiving the returned data, the terminal reads the rendering timestamp carried in the returned data and obtains the IMU data, pose data, depth boundary information or local environment change information at the current moment. It calculates the difference between the current pose and the pose corresponding to the rendering timestamp, and performs pose compensation, 2D reprojection, depth-assisted 3D reprojection, display position correction or local masking edge refresh on the virtual layer, compositing layer or local masking area.

[0171] Through this step, the system can perform edge correction on the display results under conditions of network latency, encoding / decoding latency, rapid head turning, dynamic occlusion changes, or transmission jitter, thereby reducing misalignment and jitter between virtual content and the physical environment.

[0172] S8, Semantic Security Gateway and Physical Control Closed Loop

[0173] When the candidate intent involves physical device control, medium-to-high risk output, unauthorized operation, public space impact, or abnormal triggering, the system enters the semantic security gateway process.

[0174] The semantic security gateway performs risk scoring, permission verification, confirmation control, security envelope limiting, fallback control, and audit logging on candidate intents. The risk score can be determined based on action type, action magnitude, target device level, scene context, user identity, device status, environmental status, anomaly detection results, or historical behavior records. The system selects a pass, weak confirmation, strong confirmation, denial, or downgrade strategy based on the risk score.

[0175] Once the risk scoring, permission verification, and confirmation control meet the security policy requirements, the virtual-to-physical interaction gateway transforms semantic actions, XR spatial events, or interaction intentions into physical parameter vectors or device control commands. The physical parameter vector may include at least one of the following: target device identifier, action type, amplitude, direction, duration, rate of change, start time, stop time, range of action, priority, or security restriction parameters.

[0176] The system determines the target device set based on spatial location, geofencing, device topology, device status, scene rules, or priority policies, and sends control commands to the target devices through protocol adaptation and delivery units. After the target device or execution object completes execution, it returns a device receipt or execution receipt. The system receives at least one of the following: execution status, fault status, limit status, temperature status, current status, duty cycle status, position status, remaining execution time, exception code, or current status, and writes the candidate intent, risk score, confirmation result, security envelope parameters, control command, device receipt, execution result, and timestamp into the audit log.

[0177] Through the above steps S1 to S8, this invention forms a complete methodological chain from multimodal input, predictive intent prediction, resource preheating, asset matching, spatial reconstruction, stable edge presentation, to secure physical control. This method can be used for XR spatial content presentation, as well as for virtual-real interaction scenarios requiring the linkage of physical devices, digital twin execution objects, virtual execution objects bound to physical space, or spatial event executors. It can operate within an edge-cloud collaborative architecture, or its functions can be split among terminals, edge computing, cloud computing, local servers, in-vehicle computing devices, or private network computing nodes depending on computing power, network conditions, and security requirements.

[0178] 4. Key Branch Process

[0179] 4.1 Predictive Intent Prediction and Resource Preheating

[0180] In one implementation, before the user completes their natural language input, the system performs phased parsing of the received speech segments, text prefixes, or multimodal input segments, and generates a candidate intent set by combining this parsing with current spatial awareness information. The spatial awareness information may include at least one of the following: the user's current pose, gaze direction, gesture direction, target physical entity, depth boundary, semantic boundary, scene rules, or historical interaction state.

[0181] like Figure 2 As shown, before the natural language input stream is completed, the system can generate a set of candidate intents based on the results of sliding window parsing or prefix parsing, and start resource prefetching, cache preheating, rendering preparation or computing power pre-allocation when the preset trigger conditions are met, so that the resource preparation process and the input process overlap at least partially in time.

[0182] The terminal-side subsystem can perform lightweight feature extraction on voice input, text input, and spatially aware data to form an intent prefix packet. The intent prefix packet may include at least one of the following: voice segment features, text prefix features, candidate verb prefixes, candidate object cues, environmental entity summaries, target entity categories, coarse spatial coordinates, key geometric cues, device pose summaries, confidence estimates, or noise estimates. After receiving the intent prefix packet, the edge-side subsystem or cloud-side subsystem performs multimodal fusion inference on the candidate intents and outputs a set of candidate intents and their confidence distribution.

[0183] In one implementation, the candidate intent set is defined as follows:

[0184]

[0185] in, Indicates the first One candidate intent. The system determines the intent based on the deadline. Calculate the confidence score for each candidate intent based on the received natural language prefixes, multimodal perception information, and environmental context:

[0186]

[0187] in, Indicates the end time The received speech segments, text prefixes, or multimodal input segments, Indicates time The corresponding environment context.

[0188] In one implementation, the preset trigger condition can be expressed as:

[0189]

[0190] in, This indicates a pre-set confidence threshold. When a candidate intent in the candidate intent set meets the above conditions, the system triggers resource prefetching, cache warm-up, rendering preparation, or computing power pre-allocation associated with that candidate intent.

[0191] In other implementations, the preset triggering condition may also be determined by at least one of the following: the cumulative confidence of Top-K candidate intents, the increment of candidate intent confidence, the matching score between candidate intents and environmental context, the continuous stable time of candidate intents, the user's gaze dwell time, the stability of gesture pointing, or preset rule conditions.

[0192] When preset trigger conditions are met, the system can perform at least one of the following in advance: digital asset index prefetching, candidate asset metadata prefetching, geometric prior prefetching, masking operator prefetching, interactive script prefetching, device topology information prefetching, candidate asset cache warm-up, low-definition preview preparation, rendering context creation, masking calculation task preparation, encoding pipeline preparation, GPU time slice reservation, video memory quota reservation, concurrency quota reservation, or generative model call quota reservation.

[0193] In this way, the system completes at least part of the candidate resource preparation before the user finishes natural language input, so that the natural language input process and the resource preparation process overlap at least partially in time, thereby reducing the waiting time caused by the user completing the instruction and then starting semantic understanding, asset retrieval, rendering preparation or generation call.

[0194] The intent prefix packet is not limited to a fixed-format data packet; it can also be represented as an intermediate semantic state output by the edge model, a candidate action vector, a spatial target summary, a prediction context, a candidate intent embedding vector, or a model hidden state summary. As long as it can express the candidate intent or candidate spatial target before the natural language input is completed and trigger resource preparation, spatial reconstruction, or virtual-real interaction control, it can be used as an equivalent implementation of the intent prefix packet described in this invention.

[0195] The above formulas and triggering conditions are only used to illustrate optional implementation methods and do not constitute a limitation on the candidate intent judgment method or the resource preheating triggering method.

[0196] 4.2 Arbitration of Computing Power Competition and Preheating Budget

[0197] In scenarios involving multiple users in the same domain, multiple terminals concurrently, or multiple candidate intents triggered simultaneously, the system may receive a large number of resource prefetching, cache warm-up, rendering preparation, generative engine calls, or encoding transmission requests in a short period of time. To prevent cloud or edge resources from being occupied by low-value warm-up requests, this invention sets up a computing power contention arbitration and budget control process.

[0198] In one implementation, let the current preheating request set be:

[0199]

[0200] in, Indicates the first Each preheating request is recorded. The system establishes a request record for each preheating request. The request record may include at least one of the following: request source, candidate intent, candidate asset type, spatial location, geofence, service quality level, business level, device capacity, historical hit rate, authorization status, payment status, estimated request cost, estimated resource usage, current resource availability, scene congestion level, or security level.

[0201] In one implementation, the system can calculate a comprehensive weight for each preheating request:

[0202]

[0203] in, Indicates geographical priority. Indicates the level of service quality. Indicates the business level. Indicates equipment capability. Indicates historical hit rate. Indicates whether the account is authorized or paid. Indicates the estimated cost of the request. Indicates the current resource balance or resource availability. to This indicates the preset weighting coefficient.

[0204] The system can be based on comprehensive weights The preheating requests are sorted and allocated GPU time slices, video memory quotas, concurrency quotas, cache quotas, generative model call quotas, encoding bandwidth, or edge computing resources. The comprehensive weight can also be used to determine whether a request enters full-link preheating, partial preheating, low-resolution preview preparation, semantic confirmation, contour placeholder, delayed preheating, rejection of preheating, or safe rollback.

[0205] When resources are sufficient, the system can perform full-link preheating for high-priority requests, including asset index prefetching, candidate asset loading, video memory reservation, rendering context creation, mask calculation preparation, and encoding pipeline preparation. When resources are insufficient, the system processes requests according to a tiered degradation strategy. This tiered degradation strategy includes at least one of the following: degrading from full-link preheating to index prefetching and video memory reservation; degrading from index prefetching and video memory reservation to index prefetching only or low-resolution preview preparation; degrading from low-resolution preview preparation to semantic verification or contour placeholders; and rejecting preheating or entering a local safety rollback when the risk is too high or resources are severely insufficient.

[0206] Upon completion of the preheating request, user cancellation of the request, candidate intent miss, preheating timeout, rendering completion, generation completion, or session termination, the system releases the corresponding cache, video memory, GPU time slice, concurrency quota, model call quota, or encoding bandwidth, and updates the prediction hit rate, preheating hit rate, average latency, resource usage, failure rate, degradation count, generation cost, cache hit rate, or security interception results. These statistical results can be used for subsequent priority calculations, resource scheduling strategy updates, or asset matching strategy optimization.

[0207] Through this process, the system can upgrade predictive resource preheating from a single function call to a platform-level resource scheduling mechanism under conditions of multiple users, multiple requests, and multiple resource pools. This avoids the waste of computing power caused by blind preheating and controls the computing power cost on the edge or cloud side while ensuring a low-latency experience.

[0208] The above comprehensive weighting formula is only used to illustrate optional implementation methods and does not constitute a limitation on the computing power arbitration method. In other implementation methods, the system may also use priority queues, rule tables, state machines, reinforcement learning scheduling models, prediction models, heuristic algorithms, or manual configuration strategies to perform computing power competition arbitration and preheating budget control.

[0209] 4.3 Dual-track resource routing and asset loss events

[0210] In one implementation, the system performs candidate digital asset retrieval and matching scoring in the digital asset index based on the candidate intent set and environmental context, and selects the retrieval rendering path or missing event path according to the matching score.

[0211] like Figure 3 As shown, the system switches between the retrieval rendering path and the missing event path based on the matching score between the candidate digital assets and the candidate intent.

[0212] The digital asset index can store 3D models, materials, textures, animations, special effects operators, masking operators, interaction scripts, spatial events, physical control operators, or combinations thereof. Each digital asset can be associated with semantic tags, geometric priors, adaptation rules, occlusion rules, interaction rules, licensing information, cost parameters, version number, source identifier, quality level, or applicable scenarios.

[0213] In one implementation, the candidate digital asset set is defined as follows:

[0214]

[0215] in, Indicates the first A candidate digital asset. The system can base its decisions on candidate intents. Context The matching score for candidate digital assets is calculated based on the target entity category, spatial target location, occlusion requirements, interaction requirements, authorization status, and cost parameters.

[0216]

[0217] in, Indicates semantic matching degree. Indicates geometric matching degree, Indicates the degree of matching between scene context, Indicates occlusion or composite fit. Indicates the availability of authorization. Indicates the cost of invocation or generation. to This indicates the preset weighting coefficient.

[0218] When the following conditions are met:

[0219]

[0220] When the following conditions are met, the system enters the retrieval and rendering path; when the following conditions are met:

[0221]

[0222] Or, if the digital assets, masking operators, interactive scripts, physical control operators, or adaptation rules required by the candidate intent are missing, unavailable, insufficiently authorized, or of insufficient quality, the system enters the missing event path. This indicates the scoring threshold.

[0223] During the retrieval and rendering process, the system loads the target digital asset and its geometric priors, materials, textures, animations, effects operators, masking operators, interactive scripts, or physical control operators, and sends them into the spatial reconstruction and occlusion compositing workflow. The system can also select asset versions with different precision, resolutions, or levels of interactivity based on the target device's capabilities, network status, display resolution, user permissions, or authorization status.

[0224] In the missing event path, the system generates an asset missing event signal. The asset missing event signal may include at least one of the following: missing type, candidate intent, environmental context, target entity category, required asset type, required masking type, required interaction script type, required physical control operator type, authorization status, generation requirements, cost budget, or quality requirements.

[0225] The system can perform at least one of the following based on asset missing event signals: low-cost preview, outline placeholder, semantic prompts, user authorization request, administrator authorization request, asset authorization request, generation confirmation, generative engine call, hierarchical generation, format conversion, quality check, geometric prior registration, semantic tag registration, authorization information registration, cost parameter registration, version number registration, generation result backfilling, or version governance.

[0226] The digital asset index library is not limited to an explicitly stored 3D model library, but may also include a parametric asset template library, a set of callable operators for generative models, asset representations that can be generated in real time by generative models, spatial event scripts, interaction scripts, masking operators, or physical control operators. The matching score is not limited to the above-mentioned linear weighted score, but may also come from vector similarity, ranking models, classification models, knowledge graph matching, rule engines, manual review, or multi-model fusion results.

[0227] Through the aforementioned dual-track resource routing, the system can quickly enter the retrieval and rendering path when an asset is matched. When an asset is not matched or the matching is insufficient, the asset gap can be transformed into a manageable, generateable, backfillable, and reusable system event, thereby enabling the present invention to have the digital asset supply capability for spatial computing content platforms.

[0228] The above matching scoring formula and path selection conditions are only used to illustrate optional implementation methods and do not constitute a limitation on asset scoring methods, path selection methods or asset pool forms.

[0229] 4.4 Spatial geometric fitting, pixel-level occlusion, and end-weighted projection

[0230] In one implementation, after determining the target digital asset, the system generates a physical environment representation based on physical environment geometric information, depth information, semantic boundary information, or geometric priors of the target physical entity.

[0231] like Figure 4 As shown, the system generates physical environment representations based on depth fields, semantic boundaries, meshes, point clouds, or geometric priors, and performs spatial transformations, local deformations, mask generation, pixel-level occlusion determination, and edge fusion on the target digital assets.

[0232] The physical environment characterization can be expressed as:

[0233]

[0234] in, Indicates time depth field Represents a mesh or point cloud representation. Represents semantic boundaries or the outline of a target entity. It represents the geometric priors of physical entities or scene calibration information.

[0235] The system spatially aligns the target digital assets with the physical environment representation. This spatial alignment may include at least one of coordinate transformation, scale adjustment, pose adjustment, boundary alignment, deformation, clipping, resampling, or local mesh deformation. Through this process, the target digital assets can be adapted to the spatial location, geometric boundaries, and viewing angle of real physical entities, giving the virtual content scale and boundary relationships consistent with real space.

[0236] In one implementation, the spatial alignment can be obtained by minimizing the geometric fitting error. Let the vertex set of the target digital asset be... The set of boundary points of the target entity in the physical environment representation is The depth field is The spatial transformation parameters are The local deformation parameters are The fitting parameters for the target digital asset can then be determined as follows:

[0237]

[0238] in, Represents the optimal spatial transformation parameters. This represents the optimal local deformation parameter. This represents the projection function that projects three-dimensional points onto the display plane. This represents the distance from the projection point to the set of boundary points of the target entity. Indicates virtual content in pixels The depth at that location Indicates the depth of the physical environment. This represents the pixel region involved in the fitting process. This represents the deformation smoothing constraint term. and This represents the weighting coefficient.

[0239] In this way, the system can simultaneously constrain the boundary fitting error, depth consistency and deformation continuity of the target digital asset, so that the target digital asset can be adapted according to the contour, depth and local surface changes of the real physical entity.

[0240] The above geometric fitting formula is only used to illustrate optional implementation methods and does not constitute a limitation on the spatial alignment algorithm.

[0241] In one implementation, the system can perform pixel-level occlusion determination based on virtual content depth, physical environment depth, semantic boundaries, masking data, or geometric priors. The pixel-level occlusion determination can be expressed as:

[0242]

[0243] in, Indicates the pixel coordinates in the display plane. Indicates virtual content in pixels The depth at that location Indicates the physical environment at the pixel level The depth at that location Indicates occlusion tolerance. This indicates the occlusion determination result for whether the virtual content at this pixel is displayed.

[0244] In one implementation, the occlusion tolerance It can be dynamically adjusted based on depth noise, pose changes, and network latency. The occlusion tolerance can be expressed as:

[0245]

[0246] in, Indicates time Dynamic occlusion tolerance, Indicates the basic occlusion tolerance. This represents the depth noise estimate. This represents the pose difference between the rendering time and the current time. This refers to parameters representing network jitter, backhaul latency fluctuations, or encoding / decoding latency fluctuations. , , This indicates the preset weighting coefficient.

[0247] When depth noise increases, terminal turns rapidly, or network backhaul jitter increases, the system can increase the dynamic occlusion tolerance to reduce frequent occlusion boundary jumps, white edges, or artifacts; when depth noise, attitude changes, and network jitter decrease, the system can decrease the dynamic occlusion tolerance to improve occlusion boundary accuracy and virtual-real fit accuracy.

[0248] The above dynamic occlusion tolerance formula is only used to illustrate optional implementation methods and does not constitute a limitation on the occlusion tolerance adjustment method. In other implementation methods, the system can also adjust the occlusion tolerance through regularization, model prediction, or state machine adjustment based on changes in illumination, changes in foreground occlusion, target entity movement speed, depth confidence, semantic boundary confidence, or historical occlusion stability.

[0249] For boundary areas, the system can perform feathering, anti-aliasing, depth transition, color transition, brightness transition, or local transparency adjustment to reduce white edges, jagged edges, glitches, drift, jitter, or patchiness at the edges of virtual content.

[0250] In one implementation, the spatial reconstruction and occlusion compositing process can output return frames, layered rendering results, compositing layers, mask data, depth auxiliary information, local refresh parameters, or reprojection parameters. The output results can carry a rendering timestamp for subsequent pose compensation and reprojection on the terminal side.

[0251] like Figure 5 As shown, after receiving the return frame, layered rendering result, compositing layer, mask data or reprojection parameters, the terminal subsystem calculates the pose difference based on the rendering timestamp and the current pose, and selects two-dimensional reprojection, depth-assisted three-dimensional reprojection or local mask refresh according to the pose change, depth-assisted information and local occlusion change.

[0252] After receiving the returned data, the terminal-side subsystem can read the rendering timestamp carried in the returned data. and get the current time Calculate the pose difference between the rendering time and the current time using IMU data, pose data, or localization data:

[0253]

[0254] in, This indicates the terminal pose corresponding to the rendering timestamp. This indicates the current terminal pose. The terminal-side subsystem uses this pose difference... Perform two-dimensional reprojection, depth-assisted three-dimensional reprojection, display position correction, or local masking edge refresh on virtual layers, composite layers, or local masking areas.

[0255] In other implementations, the terminal can also locally refresh the received masking data based on the latest depth boundary, the latest semantic boundary, the latest local environment image, or the latest target entity outline. For areas that have not changed significantly, the terminal can reuse the original transmission results; for areas with dynamic occlusion, edge offset, or foreground insertion, the terminal can update only the local mask edges or the local compositing layer, thereby reducing the frequency of re-requesting the cloud or edge side for complete rendering.

[0256] The spatial geometry fitting, mask generation, and pixel-level occlusion synthesis are not limited to traditional depth masking methods, but can also be achieved through semantic segmentation models, neural rendering models, optical flow compensation models, multi-layer synthesis models, SLAM relocalization, planar constraints, point cloud registration, or combinations thereof.

[0257] Through the above-mentioned spatial geometry fitting, pixel-level occlusion synthesis, and end-side projection process, the system can maintain a relatively stable relationship between virtual content and real space under scenarios such as network latency, encoding and decoding latency, user fast head turning, target entity movement, foreground occlusion changes, or lighting changes, thereby improving the realism and continuity of the XR spatial reconstruction output.

[0258] The above environmental characterization, occlusion determination, and attitude difference calculation formulas are only used to illustrate optional implementation methods and do not constitute a limitation on spatial reconstruction, occlusion synthesis, or end-side compensation algorithms.

[0259] 4.5 Semantic Security Gateway and Physical Control Closed Loop

[0260] In one implementation, when a candidate intent involves physical device control, medium-to-high risk output, unauthorized operation, public space impact, or abnormal triggering, the system does not directly issue device control commands, but instead first enters the semantic security gateway process. The semantic security gateway is used to perform risk scoring, permission verification, confirmation control, security envelope limiting, rollback control, and audit evidence storage for the candidate intent.

[0261] like Figure 6 As shown, the semantic security gateway switches between states of allowing, weak confirmation, strong confirmation, rejection, downgrading, or security rollback based on risk scoring, permission verification, and confirmation results. After meeting the security policy, it enters the security envelope limiting, device mapping, protocol adaptation, device receipt, and audit evidence storage process.

[0262] The system can calculate a risk score for candidate intentions:

[0263]

[0264] in, Indicates the risk of the action type. Indicates the risk of motion amplitude. Indicates the risk level of the target equipment. Indicates the risk in the context of the scenario. This indicates a risk to user permissions. Indicates environmental status risk. Indicates the risk of abnormal detection. to This indicates the preset weighting coefficient.

[0265] The system can be based on risk scores The system can choose from a pass, weak confirmation, strong confirmation, rejection, or downgrade strategy. In one implementation, when the risk score is below a first risk threshold, the system can directly pass or perform weak confirmation; when the risk score is between the first and second risk thresholds, the system can trigger strong confirmation; when the risk score is above the second risk threshold, or when permission verification fails, device status is abnormal, spatial area does not match, or geofence is not satisfied, the system can refuse execution, downgrade execution, or enter a security rollback process.

[0266] The strong confirmation may include at least one of the following: voiceprint confirmation, iris recognition, device binding confirmation, administrator review, physical button confirmation, time window secondary confirmation, or other confirmation methods.

[0267] Before executing physical device control, the system can also load a safety envelope based on the target device type and scenario rules. Let the original control parameter vector be:

[0268]

[0269] in, It can represent any parameter among amplitude, duration, rate of change, rotational speed, temperature, brightness, duty cycle, voltage, displacement, effective range, or starting frequency. The safety envelope can be represented as:

[0270]

[0271] as well as:

[0272]

[0273] in, and They represent the first The lower and upper limits of each control parameter, Indicates the first The maximum rate of change of each control parameter. When the control parameter exceeds the safety envelope, the system can perform limiting, truncation, smoothing, delay, segmented execution, refusal to execute, or safe rollback.

[0274] Once the semantic security gateway confirms that the candidate intent meets the security policy, the virtual-to-physical interaction gateway converts the semantic action, XR spatial event, or interaction intent into a physical parameter vector and determines the target device set based on spatial location, geofence, device topology, device status, scene rules, or priority policies. When multiple similar devices exist in the same space, the system can select the target device based on distance, orientation, region, geofence, device availability, device priority, or security level, preventing control commands from being sent to the wrong device, the wrong space, or the wrong area.

[0275] like Figure 7 As shown, the virtual-to-physical interaction gateway determines the target device set from multiple candidate devices based on spatial location, geofence, device topology, device status, and priority policy. It then sends control commands to the corresponding physical devices through the protocol adaptation and distribution unit and forms a closed-loop feedback after receiving device acknowledgments.

[0276] The protocol adaptation and delivery unit converts the physical parameter vector into protocol instructions recognizable by the target device and sends them to the target device. The protocol is not limited to IoT protocols, but may also include vehicle bus protocols, industrial bus protocols, robot control interfaces, unmanned system control interfaces, private network control protocols, standard protocols, proprietary protocols, or device control abstractions for space computing platforms. In some embodiments, the physical device control process can also be combined with spatial event presentation, digital twin object state updates, or virtual content state updates bound to the physical space, ensuring consistency between physical device execution results, spatial event display results, and audit logs.

[0277] After the target device or execution object completes execution, it returns a device receipt or execution receipt. The receipt may include at least one of the following: execution success status, execution failure status, fault status, limit status, temperature status, current status, duty cycle status, position status, remaining execution time, exception code, or current status. Based on the receipt, the system determines whether the physical control or spatial event execution is complete, whether further execution is needed, whether rollback is needed, whether an alarm is needed, or whether control commands need to be re-planned.

[0278] The audit evidence storage unit records at least one of the following: candidate intent, risk score, permission verification result, confirmation result, security envelope parameter, physical parameter vector, device control command, device receipt, execution result, abnormal status, timestamp, user identifier, device identifier, or spatial location identifier. These audit records can be used for subsequent security tracing, liability determination, operational statistics, policy optimization, or compliance review.

[0279] Through the aforementioned semantic security gateway and physical control closed loop, the system can transform natural language intents or XR space events into controlled, verifiable, rollbackable, and auditable physical execution processes, thereby reducing security risks caused by voice misrecognition, unauthorized triggering, malicious triggering, device miscontrol, or misoperation in public spaces.

[0280] The risk scoring and security envelope formulas described above are only for illustrating optional implementation methods and do not constitute a limitation on the security gateway or physical control method.

[0281] 4.6 Equivalent Implementation

[0282] To avoid the invention being circumvented by simply replacing intermediate modules, deployment locations, or data representation formats, the following technical forms can be used as equivalent implementations in different embodiments of the invention.

[0283] First, regarding the equivalent implementation of the intent prefix packet. The intent prefix packet is not limited to data packets with fixed fields, nor is it limited to being explicitly encapsulated by the terminal and sent to the cloud or edge. It can be represented as an intermediate semantic state, candidate action vector, spatial target summary, candidate intent embedding vector, prediction context, model hidden state summary, or multimodal feature cache output by a large on-device model, edge model, or cloud model. As long as it is used to express candidate intents, candidate spatial targets, or candidate actions before natural language input is completed, and triggers resource preparation, spatial reconstruction, or virtual-real interaction control, it constitutes an equivalent implementation of the intent prefix packet described in this invention.

[0284] Second, regarding the equivalent implementation of the digital asset library. The digital asset index library is not limited to an asset library that explicitly stores 3D models, materials, textures, or effects files; it can also include a parametric asset template library, a set of callable operators for generative models, asset representations that can be generated in real-time by generative models, a masking operator library, an interaction script library, a spatial event script library, or a physical control operator library. As long as the system can perform asset retrieval, generation, backfilling, reuse, or version governance based on candidate intents and environmental context, it constitutes an equivalent implementation of the digital asset and governance subsystem described in this invention.

[0285] Third, regarding the equivalent implementation of matching scoring and path selection. The matching score is not limited to a linear weighted formula, nor is it limited to explicit numerical scoring. It can be generated by vector similarity, ranking models, classification models, knowledge graph matching, rule engines, manual review, multi-model fusion results, threshold determination, priority tables, or state machines. As long as the system selects between the retrieval rendering path and the missing event path based on the matching relationship between candidate intent and candidate assets, it constitutes an equivalent implementation of the dual-track resource routing described in this invention.

[0286] Fourth, regarding the equivalent implementation of spatial reconstruction and occlusion synthesis. The spatial geometry fitting, mask generation, and pixel-level occlusion synthesis are not limited to traditional depth maps, binary masks, or explicit mesh methods. They can also be achieved through semantic segmentation models, neural rendering models, optical flow compensation models, multi-layer synthesis models, SLAM relocalization, point cloud registration, planar constraints, implicit field representation, or edge-side local refresh methods. As long as the virtual content can be adapted to the display based on the geometric boundaries, depth relationships, semantic boundaries, or dynamic occlusion relationships of the real space, it constitutes an equivalent implementation of the spatial reconstruction and occlusion synthesis described in this invention.

[0287] Fifth, regarding the equivalent implementation of the deployment architecture. The described edge-cloud collaborative architecture is not limited to a fixed three-layer deployment; it can also employ independent execution on the edge, edge-cloud collaboration, edge-edge collaboration, edge-cloud collaboration, centralized execution in the cloud, in-vehicle computing execution, execution on local high-performance computing devices, execution on dedicated network servers, or multi-node distributed execution. Regardless of whether edge-side model direct inference, edge model collaborative inference, cloud model centralized inference, or edge-cloud hybrid inference is used, as long as resource preparation, spatial reconstruction, asset generation, or virtual-real interaction control is triggered based on candidate intents with incomplete input, it falls under the equivalent deployment method of this invention.

[0288] Sixth, regarding the equivalent implementation of physical control objects and protocol adaptation. The virtual-to-physical interaction gateway is not limited to controlling a single IoT device, nor is it limited to using a single IoT protocol. Its physical control objects may include lighting equipment, fan equipment, audio equipment, mechanical structures, vehicle-mounted equipment, industrial actuators, robots, unmanned vehicles, drones, or other controllable devices associated with spatial location, geofencing, or device topology; its protocols may include IoT protocols, vehicle bus protocols, industrial bus protocols, robot control interfaces, unmanned system control interfaces, private network control protocols, standard protocols, proprietary protocols, or device control abstract interfaces for spatial computing platforms. In some implementations, physical control results can also be combined with spatial event presentation, digital twin object state updates, or virtual content state updates bound to physical space. As long as the system can convert semantic actions, XR spatial events, or interactive intentions into executable parameters and form security checks, execution feedback, or audit records, it constitutes an equivalent implementation of the virtual-to-physical interaction gateway described in this invention.

[0289] Through the above equivalent implementation, the scope of protection of this invention is not limited to a single terminal, a single cloud, a single asset library, a single occlusion algorithm, or a single device protocol, but covers the underlying gateway for spatial computing virtual-real interaction formed by candidate intent prediction, resource preparation, digital asset governance, spatial reconstruction, stable presentation on the terminal side, security control, and virtual-real to physical execution closed loop before natural language input is completed.

[0290] 5. Verifiability Statement

[0291] In one embodiment, the XR spatial reconstruction and virtual-physical interaction gateway system of the present invention can generate operation records, status records, judgment records, or audit records for key processing nodes to prove that the system has indeed performed predictive intent prediction, resource preheating, dual-track resource routing, spatial reconstruction, edge-side reprojection, security control, and virtual-physical to physical execution closed loop. These records can be stored on the terminal side, edge side, cloud side, digital asset governance platform, semantic security gateway, virtual-physical to physical interaction gateway, or independent audit and evidence storage module.

[0292] 5.1 Verifiable signals for predictive intent and resource preheating

[0293] During the predictive intent prediction phase, the system can record the start time of natural language input, the received speech segments or text prefixes, the time of intent prefix packet generation, the candidate intent set, the confidence level of the candidate intent, the preset trigger conditions, the trigger judgment result, the start time of resource preheating, and the time of natural language input completion.

[0294] By comparing the start time of resource preheating with the completion time of natural language input, it can be verified whether the system initiates resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation before the user's input is completed. If the start time of resource preheating is earlier than the completion time of natural language input, engineering evidence can be formed that the resource preparation process and the natural language input process at least partially overlap.

[0295] In one implementation, the system can also record changes in candidate intent confidence, cumulative confidence of Top-K candidate intents, confidence increment, environmental context matching results, or rule triggering results to prove that resource preheating is not started randomly, but is triggered based on candidate intents and spatial context.

[0296] 5.2 Verifiable signals for computing power competition arbitration and preheating budget

[0297] During the computing power competition arbitration phase, the system can record the source of the preheating request, request time, candidate intent, target spatial location, geofence, service quality level, business level, equipment capacity, historical hit rate, authorization status, request cost estimate, resource usage estimate, and current resource balance.

[0298] The system can also record comprehensive weights, priority sorting results, GPU time slice allocation results, video memory quota allocation results, concurrency quota allocation results, cache quota allocation results, generative model call quota, encoding bandwidth allocation results, and edge computing resource allocation results.

[0299] When resources are insufficient, the system can record the reasons for tiered degradation, degradation level, pre-degradation processing strategy, post-degradation processing strategy, rate limiting results, rejection of preheating results, or safety rollback results. These records can verify whether the system has platform-level multi-user concurrent scheduling capabilities and whether predictive resource preheating is managed in a budgeted manner.

[0300] 5.3 Verifiable signals of dual-track resource routing and asset loss events

[0301] During the digital asset matching phase, the system can record candidate intent, environmental context, target entity category, candidate digital asset set, candidate asset semantic label, geometric prior, authorization status, cost parameters, matching score, scoring threshold, and path selection result.

[0302] When the system enters the retrieval rendering path, it can record the selected target digital asset, asset version number, asset source, loading time, geometric prior call record, masking operator call record, interactive script call record, or physical control operator call record.

[0303] When the system enters the missing event path, it can record the asset missing event signal, missing type, required asset type, required mask type, required interactive script type, required physical control operator type, authorization request, generation confirmation, generative engine call time, generation result, quality check result, format conversion result, geometric prior registration result, semantic tag registration result, version number, backfill time, and asset index update record after backfill.

[0304] The above records verify that the system does not simply return failure results, but transforms asset deficiencies into manageable, generateable, backfillable, and reusable system events.

[0305] 5.4 Verifiable signals for spatial reconstruction, occlusion synthesis, and end-side reprojection

[0306] During the spatial reconstruction phase, the system can record target physical entities, depth fields, semantic boundaries, meshes, point clouds, plane detection results, physical entity geometric priors, physical environment representations, target digital assets, geometric fitting parameters, deformation parameters, clipping parameters, resampling parameters, and spatial alignment results.

[0307] During the occlusion compositing stage, the system can record the mask type, mask generation time, pixel-level occlusion judgment result, occlusion tolerance, edge blending parameters, anti-aliasing parameters, depth transition parameters, color transition parameters, brightness transition parameters, and compositing output time.

[0308] During the end-side reprojection stage, the system can record the rendering timestamp of the returned frame or compositing layer, the terminal receiving time, the current IMU data, the current pose data, the pose corresponding to the rendering timestamp, the current pose, the pose difference, the reprojection trigger time, the reprojection method, the local mask refresh area, and the refresh result.

[0309] The above records can be used to verify whether the system performs virtual-real alignment based on the geometric boundaries, depth relationships, and dynamic pose changes of real space, rather than simply performing two-dimensional overlay.

[0310] 5.5 Verifiable signals for semantic security gateway and physical control closed loop

[0311] During the semantic security gateway phase, the system can record candidate intents, action types, action magnitudes, target device levels, scene contexts, user identities, user permissions, device status, environment status, anomaly detection results, risk scores, risk thresholds, permission verification results, and security policy selection results.

[0312] When the system triggers confirmation control, it can record the weak confirmation result, strong confirmation type, strong confirmation time, confirming device, confirming user, and confirmation success or failure result. The strong confirmation type can include at least one of voiceprint confirmation, iris confirmation, device binding confirmation, administrator review, physical button confirmation, or time window secondary confirmation.

[0313] During the safety envelope limiting phase, the system can record the target device type, original control parameters, parameter upper limit, parameter lower limit, rate of change limit, limiting processing result, smoothing processing result, segmented execution strategy, reason for refusal to execute, or safety rollback result.

[0314] During the physical control closed-loop phase, the system can record physical parameter vectors, target device sets, geofencing determination results, device topology matching results, protocol adaptation results, control commands, issuance time, device acknowledgments, execution status, exception codes, execution completion time, and rollback processing results.

[0315] The above records can be used to verify whether natural language intent or XR space events have undergone risk scoring, permission verification, security envelope and device feedback closed loop, thereby proving that the physical control process is controllable, traceable and auditable.

[0316] 5.6 Audit Records and the Formation of the Chain of Evidence

[0317] In one implementation, the system can uniformly write the aforementioned key node records into the audit log. The audit log may include at least one of the following: session identifier, user identifier, terminal identifier, spatial location identifier, timestamp, candidate intent, trigger condition, resource preheating record, computing power allocation record, asset matching record, missing event record, generation backfill record, rendering timestamp, posture difference, reprojection record, risk score, security envelope parameter, device control command, device receipt, and execution result.

[0318] like Figure 8 As shown, during the processes of input prefix encapsulation, semantic reasoning and resource preheating, digital asset governance, spatial reconstruction presentation, and security control and auditing on the terminal side, the system sequentially generates and transmits intent prefix packets, candidate intent sets, preheating request records, asset missing event signals, spatial reconstruction output results, physical parameter vectors, device receipts, and audit records.

[0319] The audit logs can be stored in at least one of the following ways: local logs, edge logs, cloud logs, hash digests, timestamp signatures, permission signatures, device signatures, chained records, trusted execution environment records, or third-party evidence records.

[0320] In one implementation, the system can correlate prediction trigger records, resource preheating records, asset routing records, spatial reconstruction records, edge reprojection records, security control records, and device acknowledgment records within the same session in chronological order to form a complete chain of evidence. This chain of evidence can prove that the system completed the entire process from predicting candidate intents that had not yet been fully input, to resource preparation, asset governance, spatial reconstruction, stable edge presentation, security control, and physical execution feedback within the same interaction.

[0321] In one implementation, the score, threshold, timestamp, attitude difference, security envelope parameters, control commands, execution receipts, and abnormal states in the audit log can serve as verification data to determine whether the corresponding technical steps were actually executed. The system can use this verification data to trace the execution process of predictive preheating, dual-track resource routing, spatial reconstruction, end-side reprojection, security gateway, and physical control closed loop.

[0322] The aforementioned verifiability records are not required to be generated in every implementation. Different deployment scenarios may choose to record one or more of these based on terminal capabilities, edge capabilities, cloud capabilities, security levels, privacy requirements, or compliance requirements. As long as the system can record the inputs, judgments, outputs, or feedback of key processing nodes, it can be considered an implementation of the verifiability of this invention. Specific Implementation

[0323] The following embodiments are used to further illustrate specific implementations of the present invention. It should be understood that the following embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the technical features of the various embodiments can be combined with each other.

[0324] Example 1: XR Spatial Reconstruction Guide Based on AI Glasses

[0325] In one embodiment, the user wears AI glasses or an XR terminal to enter a real physical space, such as a museum exhibition hall, a heritage park, a commercial district, a theme park, an industrial park, or an urban public space. The terminal-side subsystem acquires the user's voice input, eye movement information, head pose, IMU data, depth information, environmental images, and positioning information to form a multimodal input data stream.

[0326] When a user hasn't fully uttered a command, such as saying "Take this city wall..." or "Make the exhibits in front...", the system generates a set of candidate intents based on the received speech fragments, the user's gaze area, gesture direction, the outline of the target physical entity, and its spatial location. The system determines whether the user might want to perform candidate intents such as history restoration, information annotation, virtual character appearance, spatial animation playback, or exhibit structure disassembly. When the confidence level of the candidate intent meets preset conditions, the system initiates digital asset index prefetching, geometric prior prefetching, masking operator prefetching, and rendering context preparation in advance.

[0327] When a user says "Restore this city wall to its appearance during the Warring States period", the system retrieves 3D models, materials, textures, historical annotations, animations or masking operators related to the restoration of the Warring States city wall from the digital asset index based on the candidate intent and the target physical entity. The system then calculates a matching score based on semantic matching degree, geometric matching degree, scene context matching degree, occlusion adaptation degree, authorization status and cost parameters.

[0328] When the matching score meets the scoring threshold, the system enters the retrieval and rendering path, loads the target digital asset, and performs spatial geometry fitting based on the depth field, semantic boundaries, point cloud, or geometric priors of the on-site city wall ruins. Based on the real wall boundaries, incomplete areas, occlusion relationships, and the user's current perspective, the system performs deformation, cropping, resampling, and pixel-level occlusion synthesis on the virtual restoration content, ensuring that the virtual restored wall fits the real wall boundaries.

[0329] After rendering is completed in the cloud or at the edge, the returned frames, layered rendering results, masking data, depth assistance information, rendering timestamps, and reprojection parameters are sent to the AI ​​glasses. The terminal side compares the current IMU data and current pose with the pose corresponding to the rendering timestamp, calculates the pose difference, and performs terminal reprojection and local edge refresh on the virtual layer or local masking area, so that the restored content remains spatially stable when the user turns their head, moves forward or backward, or is briefly occluded.

[0330] In this embodiment, the system can record user speech segments, candidate intents, confidence levels, warm-up start times, input completion times, asset matching scores, target asset version numbers, spatial fitting parameters, occlusion determination results, rendering timestamps, pose differences, and reprojection records. These records can be used to verify whether the system initiates resource preparation before the natural language input is completed and completes XR spatial reconstruction based on real spatial geometric relationships.

[0331] Example 2: Asset Missing Events and Generation Backfill

[0332] In one embodiment, the user issues a natural language command in the XR terminal, such as "Make a scene of an ancient army marching above this site appear." The system generates a set of candidate intentions based on the voice input, the user's gaze direction, the target spatial location, the depth boundary, and the scene context, and determines whether the command involves virtual character groups, scene animation, ground adaptation, occlusion synthesis, and spatial event orchestration.

[0333] The system retrieves 3D models, character animations, queue paths, flag effects, dust effects, sound scripts, and masking operators related to "ancient army marching" from the digital asset index. If the system finds that existing assets cannot meet the geometric dimensions, style requirements, licensing conditions, or interaction requirements of the current site space, and the matching score is lower than the scoring threshold, it generates an asset missing event signal.

[0334] The asset missing event signal can record candidate intent, target spatial location, target entity category, required asset type, missing type, scene style, spatial boundaries, animation duration, quality requirements, authorization status, and cost budget. The system can first output a low-cost preview or outline placeholder to the terminal, such as displaying the approximate path of the queue, character silhouettes, or a semi-transparent placeholder layer above the ruins, to provide users with immediate feedback.

[0335] After user authorization or automatic system confirmation, the system invokes the generative engine to generate corresponding digital assets based on candidate intent, site spatial boundaries, historical style constraints, number of characters, movement paths, occlusion relationships, and device performance requirements. The generated results may include character models, motion animations, flag effects, ground dust effects, interactive scripts, masking operators, or spatial event scripts.

[0336] The system performs quality checks, format conversions, geometric prior registration, semantic tag registration, authorization information registration, cost parameter registration, and version number registration on the generated results, and then populates the generated results into the digital asset index library. Subsequently, when candidate intentions such as "military march," "battle formation restoration," and "historical scene outbreak" reappear in the same or similar spatial scenes, the system can reuse the asset or its parameterized version, reducing the cost of repeated generation.

[0337] In this embodiment, the system can record asset missing event signals, missing type, matching score, generative engine call time, generation prompt parameters, generation result, quality check result, asset backfill time, asset version number, and subsequent reuse records. These records demonstrate the system's ability to transform asset gaps into generateable, backfillable, and governable platform assets.

[0338] Example 3: High-concurrency arbitration for multiple users simultaneously

[0339] In one embodiment, multiple users are simultaneously located in the same XR experience space, such as a cable car cabin in a scenic area, a cruise ship deck, a museum exhibition hall, an immersive theater, a theme park queue area, or a large outdoor archaeological site. Multiple terminals continuously upload audio clips, text prefixes, gaze direction, gesture pointing, pose information, depth information, and environmental awareness information within the same time period.

[0340] Since multiple users may trigger candidate intent prediction, the system receives multiple resource prefetching, cache preheating, rendering preparation, generative model invocation, and encoding transmission requests within a short period of time. To avoid all requests simultaneously consuming GPU, video memory, cache, bandwidth, and generative model quota, the computing power contention arbitration and budget control subsystem establishes a request record for each request.

[0341] The request record may include the user's location, service quality level, business level, device capabilities, historical hit rate, authorization status, payment status, estimated request cost, estimated resource usage, current resource availability, and security level. The system sorts multiple pre-warming requests based on a comprehensive weighting, prioritizing requests with more critical location, higher service level, higher hit rate, lower cost, or higher security level.

[0342] For high-priority requests, the system performs full-chain warm-up, including asset index prefetching, candidate asset loading, video memory reservation, rendering context creation, mask calculation preparation, and encoding pipeline preparation. For medium-priority requests, the system may perform partial warm-up, such as only asset index prefetching, low-resolution preview preparation, or geometry prior prefetching. For low-priority requests, the system may perform semantic verification, contour placeholders, or delayed warm-up. In cases of severe resource shortage, excessive risk, or low request hit rate, the system may refuse warm-up or enter local safe rollback.

[0343] When a user cancels a request, the candidate intent is not hit, the warm-up times out, rendering is completed, or the session ends, the system releases the corresponding resources and updates the prediction hit rate, warm-up hit rate, average latency, resource usage, failure rate, number of degradations, generation cost, and cache hit rate.

[0344] In this embodiment, the system can record the preheating request queue, overall weight, resource allocation results, degradation strategy, resource release time, and preheating hit results. These records demonstrate that the system is not a single-terminal function call scheme, but rather a spatial computing virtual-physical interaction gateway with platform-level multi-user, multi-request, and multi-resource pool scheduling capabilities.

[0345] Example 4: Virtual-to-Physical Control Security Closed Loop

[0346] In one embodiment, the user issues natural language commands in the XR space, such as "Turn the lights ahead red and follow the character," "Turn on the wind effect in this area," "Make the seats in the carriage vibrate," or "Start the robot performance in front." The system generates candidate intents involving physical device control based on voice input, user location, target space area, device topology, geofencing, and scene rules.

[0347] When the candidate intent involves physical device control, the system first enters the semantic security gateway process. The semantic security gateway calculates a risk score based on the action type, action magnitude, target device level, scene context, user permissions, device status, environmental status, and anomaly detection results. If the risk score is low and permission verification passes, the system can allow the process or perform weak confirmation; if the risk score is in the medium range, the system triggers strong confirmation; if the risk score is too high, permissions are insufficient, the device is abnormal, or the geofence requirement is not met, the system refuses to execute or enters a security fallback.

[0348] Before executing control, the system loads a safety envelope for the target device. For example, for lighting equipment, the safety envelope can limit brightness, flashing frequency, duration, and color switching rate; for fan equipment, the safety envelope can limit wind speed, start-up frequency, and duration; for seat vibration equipment, the safety envelope can limit vibration amplitude, duration, and rate of change; and for robots or mechanical structures, the safety envelope can limit range of motion, speed, acceleration, stopping distance, and safety zone.

[0349] After the semantic security gateway confirms the connection, the virtual-to-physical interaction gateway converts the candidate intent into a physical parameter vector. The system selects the target device based on spatial location, geofencing, device topology, device status, and priority policies, and converts the physical parameter vector into control commands recognizable by the target device through the protocol adaptation and distribution unit. These control commands can be sent via IoT protocols, vehicle bus protocols, industrial bus protocols, robot control interfaces, unmanned system control interfaces, private network control protocols, or internal control protocols of the space computing platform.

[0350] After execution, the target device returns a device acknowledgment. The system uses the acknowledgment to determine whether the execution was successful, whether a limit switch was triggered, whether a fault occurred, whether a rollback is needed, or whether control commands need to be reprogrammed. If the device does not return an acknowledgment, the acknowledgment is abnormal, or the execution result exceeds the safety envelope, the system can trigger a stop, degradation, rollback, alarm, or manual review.

[0351] In this embodiment, the system can record candidate intents, risk scores, permission verification results, strong confirmation results, security envelope parameters, physical parameter vectors, target device selection results, protocol adaptation results, control commands, device receipts, exception codes, and execution results. These records demonstrate that natural language intents or XR spatial events do not directly control physical devices, but rather form a closed-loop execution process after security verification, parameter restrictions, protocol adaptation, execution feedback, and audit evidence storage.

[0352] Example 5: Linkage between physical control results and spatial event presentation

[0353] In one embodiment, while performing physical device control, the system can also update the display effects of spatial events, the state of digital twin objects, or the state of virtual content that are bound to the real physical space. For example, a user in a real exhibition hall can say, "Turn the lights in front into night mode and open a time portal," "Make a glowing path appear on the lake and simultaneously turn on the waterfront lights," or "Decompose the display structure of the objects in this showcase and adjust the showcase lights."

[0354] The system generates a set of candidate intents based on user natural language input, gaze direction, spatial location, target physical entity, environmental geometric boundaries, and scene rules, and determines whether the instruction involves both spatial presentation and physical device interaction. The system retrieves corresponding spatial event scripts, animation scripts, special effects operators, masking operators, or interaction scripts from the digital asset index, and after passing through the semantic security gateway, the virtual-to-physical interaction gateway converts the physical control portion into physical parameter vectors or device control commands.

[0355] For the spatial presentation, the system performs spatial geometry fitting, occlusion synthesis, and end-weighted projection based on the depth information, semantic boundaries, and geometric priors of the actual walls, lake surface, display cases, or space in front. For the physical equipment, the system determines the target equipment and issues control commands based on spatial location, geofencing, equipment topology, and security envelope. Spatial event display results, physical equipment receipts, and audit logs can be associated and saved within the same session.

[0356] This embodiment demonstrates that the virtual-to-physical interaction gateway of the present invention can not only control physical devices, but also ensure that the execution results of physical devices are consistent with the presentation of spatial events, the update of digital twin status, or the update of virtual content status, thereby forming a complete closed loop of virtual-physical linkage.

[0357] The above embodiments illustrate the specific implementation of the present invention from five perspectives: XR spatial presentation, asset missing generation, high-concurrency computing power scheduling, physical device security control, and the linkage between physical control results and spatial event presentation. Different embodiments can be combined according to the actual deployment environment. For example, in multi-person XR guided tours, predictive resource preheating, asset missing generation backfilling, edge-side heavy projection, and physical control security closed loop can be used simultaneously.

Claims

1. A method for XR spatial reconstruction and virtual-real interaction gateway, characterized in that, Includes the following steps: S1. Acquire a multimodal input data stream, wherein the multimodal input data stream includes at least one of voice input and text input, and at least one of spatially related visual information, pose information, depth information, inertial measurement information, eye tracking information, gesture information, or positioning information; S2. Perform sliding window parsing or prefix parsing on the natural language input stream formed by the voice input or text input, and generate a candidate intent set and its confidence distribution by combining the spatially related information; S3. Before the natural language input stream is completed, when the candidate intent set meets the preset triggering conditions, the prefetching, cache warm-up, rendering preparation or computing power pre-allocation of cloud resources, edge resources or local resources associated with the candidate intent is triggered. S4. Based on the candidate intent set and the environmental context, perform asset matching scoring in the digital asset index library, and select the retrieval rendering path or missing event path according to the scoring threshold. S5. In the retrieval and rendering path, the target digital asset is obtained, and geometric fitting, occlusion modeling, mask generation and pixel-level occlusion synthesis are performed in combination with physical environment geometric information, depth information or semantic boundary information to generate spatial reconstruction output results. S6. In the missing event path, generate an asset missing event signal and trigger the generative engine to generate digital assets. After generation, the digital assets are backfilled into the asset library and enter the rendering and compositing output. S7. The spatial reconstruction output result is transmitted to the terminal in a streaming form, and the terminal performs pose compensation, reprojection or local masking refresh based on the rendering timestamp and real pose to stably present the virtual-real alignment result. S8. When the candidate intent involves physical device control or is determined to be a medium-to-high risk intent, the semantic security gateway process is entered to perform risk scoring, permission verification, confirmation control, security envelope limiting and audit recording on the candidate intent. After the security policy is met, the semantic action is mapped into parameterized device control instructions and sent to the physical device associated with the spatial location or geofence. Device acknowledgments are received and a closed-loop feedback is formed.

2. The method for XR spatial reconstruction and virtual-real interaction gateway according to claim 1, characterized in that: In S2, the system performs time-slicing processing on continuous speech streams, text streams, or multimodal input streams to form semantic prefix features, and combines them with visual environment context, spatial target cues, or terminal pose summaries to generate a set of candidate intentions and their confidence distribution. The preset triggering conditions in S3 include at least one of the following: the confidence level of the candidate intent exceeds a preset threshold, the confidence level increment of the candidate intent exceeds a preset threshold, the cumulative confidence level of multiple candidate intents exceeds a preset threshold, the matching degree between the candidate intent and the environmental context exceeds a preset threshold, and the candidate intent meets the preset rule conditions. When the preset triggering conditions are met, the system initiates resource prefetching, cache preheating, rendering preparation, or computing power pre-allocation before the natural language input stream is completed, so that the resource preparation process and the natural language input process overlap at least partially in time.

3. The method for XR spatial reconstruction and virtual-real interaction gateway according to claim 1, characterized in that: In S2, the terminal performs lightweight feature extraction and segmentation on the input stream, forms an intent prefix packet, and sends it to the edge node or the cloud. The intent prefix packet includes at least one of the following: speech segment features, text prefix features, candidate verb prefixes, candidate object cues, environmental entity summaries, target entity categories, coarse spatial coordinates, key geometric cues, device pose summaries, confidence estimates, or noise estimates. Edge nodes or the cloud perform multimodal fusion reasoning based on the intent prefix packet, construct a candidate intent set and its confidence distribution, and output a set of associated asset tags, spatial target clues, or candidate mask types.

4. The method for XR spatial reconstruction and virtual-real interaction gateway according to claim 1, characterized in that: In S3, before triggering resource prefetching, cache preheating, rendering preparation or computing power pre-allocation, computing power competition arbitration and preheating budget control are performed on the preheating request; The system determines the priority or comprehensive weight of preheating requests based on at least one of the following: geographical priority, service quality level, business level, equipment capacity, historical hit rate, payment status, estimated request cost, or current resource availability. It then allocates GPU time slices, video memory quotas, concurrency quotas, or cache quotas according to the priority or comprehensive weight. When resources are insufficient, a preheating and degradation strategy is executed according to the hierarchical degradation strategy. The hierarchical degradation strategy includes at least one of the following: degradation from full-link preheating to index prefetching and video memory reservation, degradation from index prefetching and video memory reservation to index prefetching only or low-definition preview preparation, and degradation from low-definition preview preparation to semantic confirmation or placeholder prompt.

5. The method for XR spatial reconstruction and virtual-real interaction gateway according to claim 1, characterized in that: In step S4, the system retrieves candidate digital assets from the digital asset index based on the candidate intent set and the environmental context, and calculates the matching score between the candidate digital assets and the candidate intent. When the matching score is not lower than the score threshold, the retrieval rendering path is entered, the target digital asset and its geometric prior are loaded, and the mask modeling, geometric fitting, pixel-level occlusion synthesis, encoding back transmission and terminal reprojection compensation process is entered. When the matching score is lower than the score threshold, it is determined to be an asset missing event, enters the missing event path, generates an asset missing event signal, and performs at least one of the following: low-cost preview output, outline placeholder output, semantic prompts, authorization request, generative engine call, hierarchical generation, generation result backfilling, or version governance. The backfilling of the generated results includes at least one of the following: semantic tags, geometric priors, authorization information, version number, or cost parameters of the generated assets.

6. The method for XR spatial reconstruction and virtual-real interaction gateway according to claim 1, characterized in that: In S5, the system generates or updates the physical environment representation based on the depth field, semantic boundary, mesh, point cloud or physical entity geometric prior, and performs geometric fitting, deformation, clipping or resampling on the target digital asset according to the physical environment representation, so that the boundary of the target digital asset is aligned with the target physical entity in the spatial coordinate system. The system generates a dynamic mask based on the physical environment representation, and performs pixel-level occlusion determination and edge fusion on the target digital asset and the physical environment background to suppress white edges, glitches, or jitter. In step S7, after receiving the return frame, layered rendering result or composite layer, the terminal reads the rendering timestamp carried by the return data, obtains the IMU data or pose data at the current time, calculates the pose difference corresponding to the rendering timestamp, and performs two-dimensional reprojection, depth-assisted three-dimensional reprojection or local masking edge refresh on the virtual layer or composite layer to offset the drift caused by network latency, rapid head turning or dynamic occlusion.

7. An XR spatial reconstruction and virtual-real interaction gateway system for performing the method according to any one of claims 1 to 6, characterized in that, It includes at least two subsystems from the following: terminal-side subsystem, edge-side subsystem, cloud-side subsystem, digital asset and governance subsystem, spatial reconstruction and occlusion synthesis subsystem, semantic security gateway subsystem, virtual-to-physical interaction gateway subsystem, and computing power competition arbitration and budget control subsystem. The terminal-side subsystem is used to perform at least one of the following: multimodal acquisition, intent prefix packet generation, local pose prediction, display output, terminal-side re-projection, local masking refresh, or local degradation. The edge-side subsystem is used to perform at least one of the following: near-end caching, lightweight prediction, proximity rendering encoding, regional arbitration, or network outage degradation. The cloud subsystem is used to perform at least one of the following: predictive semantic prediction, resource preheating and scheduling, asset retrieval, generative engine invocation, masking or fitting calculation, cloud rendering, real-time encoding or streaming distribution. The digital asset and governance subsystem is used to support the retrieval and rendering path, missing event path, result backfilling, metadata governance, or authorization information management. The spatial reconstruction and occlusion synthesis subsystem is used to generate or update physical environment representations and perform geometric fitting, mask generation, pixel-level occlusion determination or edge fusion. The semantic security gateway subsystem is used to perform risk scoring, permission verification, strong authentication, security envelope limiting, fallback control, or audit evidence storage. The virtual-to-physical interaction gateway subsystem is used to map semantic actions into physical parameter vectors or device control commands, and send them to the corresponding physical devices based on spatial location, geofence, or device topology. The computing power competition arbitration and budget control subsystem is used to queue, prioritize, allocate budgets, limit traffic, downgrade, or reclaim resources for preheating requests.

8. The XR spatial reconstruction and virtual-real interaction gateway system according to claim 7, characterized in that: The digital asset and governance subsystem includes at least one of the following: a digital asset index library, a matching scoring and routing decision module, and a generative engine invocation and backfilling module. The digital asset index library stores vectorized indexes and metadata of 3D models, materials, special effects operators, masking operators, interactive scripts, or physical control operators. The metadata includes at least one of semantic tags, geometric priors, adaptation rules, authorization information, cost parameters, or version numbers. The matching score and routing decision module is used to calculate the matching score between candidate intents and candidate digital assets, and select the retrieval rendering path or missing event path based on the score threshold. The generative engine invocation and backfilling module is used to invoke the generative engine to generate digital assets in the missing event path and backfill the generation results to the asset library for reuse in subsequent requests. The spatial reconstruction and occlusion synthesis subsystem includes at least one of the following: an environmental geometry / semantic modeling module, a geometry fitting and deformation module, and a mask generation and edge blending module.

9. The XR spatial reconstruction and virtual-real interaction gateway system according to claim 7, characterized in that: The semantic security gateway subsystem includes at least one of the following: an intent risk scoring module, a strong authentication and access control module, a security envelope limiting and fallback module, and an audit and evidence storage module. The intent risk scoring module is used to calculate a risk score based on action type, amplitude parameter, target device level, scene context or anomaly detection result, and select a release, weak confirmation, strong confirmation, rejection or downgrade strategy based on the risk score. The safety envelope limiting and backoff module is used to impose upper bound constraints or rate of change constraints on at least one of the following: amplitude, duration, rate of change, rotation speed, temperature, brightness, duty cycle, voltage, or displacement of the equipment control command. The virtual-to-physical interaction gateway subsystem includes at least one of the following: a semantic dynamics interpreter, a device mapping and geofencing association module, a protocol adaptation and distribution module, and a closed-loop feedback module. The semantic dynamics interpreter is used to convert semantic verbs or action descriptions into physical parameter vectors and their timing control curves. The device mapping and geofence association module is used to determine the target device set based on spatial location, geofence, device topology or priority strategy. The protocol adaptation and delivery module is used to deliver device control commands and receive device acknowledgments. The computing power competition arbitration and budget control subsystem includes at least one of the following: a preheating budget allocation module, a multi-dimensional weight arbitration module, and a hierarchical degradation and rate limiting module.

10. A product for performing the method according to any one of claims 1 to 6, characterized in that, Including at least one of computer-readable storage media, programs, XR terminals, edge computing nodes, cloud service platforms, digital asset governance platforms, or virtual-to-physical interaction gateway devices; The computer-readable storage medium and program are used to cause a processor to perform the method according to any one of claims 1 to 6; The XR terminal includes at least one of the following: a sensor assembly, an intent prefix packet generation and transmission unit, a communication unit for receiving cloud or edge rendering output, a display output unit, and a pose compensation / reprojection unit based on rendering timestamps and real-time poses. The edge computing node is used to perform at least one of the following: near-end caching, lightweight prediction, near-end rendering encoding, edge arbitration, or network outage degradation. The cloud service platform is used to perform at least one of the following: candidate intent prediction, resource preheating, asset retrieval, generative engine invocation, spatial reconstruction calculation, rendering compositing, real-time encoding, or streaming distribution. The digital asset governance platform is used to perform at least one of the following: asset indexing, matching scoring, missing event management, authorization information management, result backfilling, or version governance. The virtual-to-physical interaction gateway device is used to perform at least one of the following: semantic action parameterization, device mapping, protocol adaptation, control issuance, device receipt reception, or audit recording.

Images