Multi-dimensional demonstration method and device based on time domain technology and storage medium

Abstract

This application discloses a multi-dimensional demonstration method, apparatus, and storage medium based on temporal domain technology, relating to the field of data interaction technology. This application first acquires a temporal file containing multiple viewpoints and / or multiple time points, and encapsulates multi-dimensional index information including spatial and temporal indices. Then, based on the multi-dimensional index information, a demonstration narrative stream is constructed, including a sequence of keyframes for the demonstration and the camera motion paths of the keyframes. During playback rendering, the current camera state is acquired in real time, and combined with the predicted information of the camera motion paths, the target display area within the current view frustum is calculated. Finally, the corresponding target pyramid tile data is retrieved from the multi-dimensional index information and progressively decoded and rendered in order of increasing resolution, achieving seamless loading of the demonstration scene from an overall overview to local micro-details. This improves the loading efficiency of the multi-dimensional demonstration scene and reduces resource consumption.

Description

Technical Field

[0001] This application relates to the field of data interaction technology, specifically to a multidimensional demonstration method, device, and storage medium based on time-domain technology. Background Technology

[0002] A time-domain file is a sequential data file with time as its core dimension, associated with multiple perspectives and orientations. Referring to the specific content of CN114820575A, "Image Verification Method, Apparatus, Computer Equipment and Storage Medium," for example, the three-dimensional target object mentioned in paragraphs

[0043] -

[0044] of the specification, a time-domain file can refer to a three-dimensional target object file containing time-dimensional information. A time-domain file can record the dynamic characteristics of the target object at different points in time (or time periods), and also associate information from multiple perspectives and orientations. In the time dimension, the time-domain file contains continuous or discrete time nodes, which can reflect the dynamic changes of the target object over time. In the visual dimension, time-domain data at the same point in time can correspond to different observation perspectives, recording the time-domain performance of the target object under different observation angles. In the orientation dimension, time-domain data at the same perspective and the same point in time can be further associated with different orientations.

[0003] Traditional multidimensional presentation systems face significant technical bottlenecks when processing large-scale spatiotemporal data. Traditional methods typically employ static loading or simple Levels of Detail (LOD) techniques, which usually store multi-view, multi-time-point visual data independently, lacking a unified spatiotemporal indexing framework. This distributed storage leads to inefficient data retrieval, requiring the system to traverse large amounts of irrelevant data during loading, resulting in long initial loading times and high memory usage. Especially when processing high-resolution imagery or 3D models, the data volume grows exponentially, and traditional methods often fail to effectively manage data levels, leading to unstable rendering frame rates and noticeable stuttering.

[0004] At the dynamic rendering level, traditional technologies generally lack the ability to predict the user's viewing intentions. Systems typically employ full-scene loading or fixed-area preloading strategies, requiring continuous memory resident of full-scene data regardless of camera movement. When the camera moves rapidly or browses a large scene, a large amount of redundant data outside the current view frustum is loaded into video memory, wasting memory bandwidth and crowding out data rendering resources in critical areas.

[0005] Therefore, traditional multidimensional demonstration scenarios have low loading efficiency and high resource consumption, making it difficult to meet the needs of efficient demonstration of large-scale spatiotemporal data. Summary of the Invention

[0006] The purpose of this application is to provide a multidimensional demonstration method, device and storage medium based on time-domain technology to solve the problems of low loading efficiency and high resource consumption in traditional multidimensional demonstration scenarios.

[0007] To achieve the above objectives, the first aspect of this application provides a multidimensional demonstration method based on time-domain techniques, comprising:

[0008] Obtain a time-domain file containing visual data from multiple viewpoints and / or multiple time points, and encapsulate multi-dimensional index information containing spatial and temporal indexes, wherein the multi-dimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles;

[0009] A demonstration narrative stream is constructed based on the multidimensional index information, and the demonstration narrative stream includes a sequence of keyframes of the demonstration and the camera motion path of the keyframes;

[0010] During the playback rendering process, the current camera status is acquired in real time, and the target display area within the current view frustum is calculated by combining the predicted information of the camera motion path.

[0011] Based on the predicted direction of the target display area and the camera motion path, the corresponding target pyramid tile data is retrieved from the multidimensional index information, and progressively decoded and rendered in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

[0012] A second aspect of this application provides a multi-dimensional demonstration device based on time-domain technology, comprising:

[0013] The acquisition module is used to acquire a time-domain file, which contains visual data from multiple perspectives and / or multiple time points, and encapsulates multi-dimensional index information containing spatial and temporal indexes. The multi-dimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles.

[0014] A construction module is used to construct a demonstration narrative stream based on the multidimensional index information, the demonstration narrative stream including a sequence of keyframes of the demonstration and the camera motion path of the keyframes;

[0015] The calculation module is used to obtain the current camera status in real time during the playback rendering process, and calculate the target display area within the current view frustum by combining the prediction information of the camera motion path.

[0016] The rendering module is used to retrieve the corresponding target pyramid tile data from the multidimensional index information based on the predicted direction of the target display area and the camera motion path, and to perform progressive decoding and rendering in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

[0017] A third aspect of this application provides a computer-readable storage medium storing a program that can be loaded by a processor and executed to perform the aforementioned multidimensional demonstration method based on time-domain technology.

[0018] The beneficial effects of this application are:

[0019] This application leverages multi-dimensional index information encapsulated in temporal files and its index mapping relationship with multi-resolution pyramid tiles. It eliminates the need to load the entire visual data frame, retrieving only the tile data for the target area as needed. This significantly reduces the resource consumption of data transmission and loading, lowering resource consumption for multi-dimensional presentations from the data layer. Based on multi-dimensional index information, a presentation narrative flow containing keyframe sequences and camera motion paths is constructed, enabling predictable camera motion trajectories. This provides a clear trajectory basis for the targeted retrieval of subsequent tile data, reducing aimless data loading and improving the loading specificity and overall efficiency of the presentation scene. During playback rendering, the target display area is calculated using camera motion path prediction information, and tile data is retrieved according to the predicted direction. This achieves predictive loading of tile data, reducing data waiting time during the presentation and improving the smoothness of scene loading. By adopting a progressive decoding and rendering method with resolutions ranging from low to high, low-resolution tiles are loaded quickly to form an overall overview, and then high-resolution tiles are gradually superimposed to enhance details. This achieves both rapid presentation of the demonstration scene and seamless loading from the overall to the micro-details. It improves loading efficiency while ensuring the visual experience of the demonstration, effectively solving the contradiction between loading efficiency and detail display in traditional multi-dimensional demonstrations.

[0020] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multidimensional demonstration method based on time-domain technology provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of a demonstration packet data structure and pyramid tile index for a time-domain file provided in an embodiment of this application;

[0023] Figure 3 This is a timing diagram for the synchronization control of a presentation terminal and an audience terminal provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a multidimensional demonstration device based on time-domain technology provided in the embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified. Details are set forth in the following description for illustrative purposes. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessarily obscuring the description of this application. Therefore, this application is not intended to be limited to the embodiments shown, but rather to be consistent with the broadest scope of the principles and features disclosed herein.

[0027] Figure 1 This is a flowchart illustrating a multi-dimensional demonstration method based on time-domain techniques provided in an embodiment of this application. Figure 1 As shown, this multidimensional demonstration method may include steps 101-104, which will be described in detail below.

[0028] Step 101: Obtain the temporal file. The temporal file can contain visual data from multiple perspectives and / or multiple time points, and encapsulates multidimensional index information containing spatial and temporal indices. The multidimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles.

[0029] This step is the underlying data preparation stage, which involves reading a pre-defined time-domain file from local storage, a cloud server, or other media. The time-domain file is a unified data carrier specifically designed for multi-dimensional demonstration scenarios, and is not a typical 3D model or image file. After reading, the time-domain file is first validated for legality, then its internal encapsulation structure is parsed to extract visual data, multi-dimensional index information, and their relationship with the multi-resolution pyramid tiles.

[0030] Multidimensional indexing is the core indexing system of time-domain files, comprising two core dimensions: spatial indexing and temporal indexing. The spatial index is used to locate the spatial position of multi-view visual data, while the temporal index is used to locate the time nodes of multi-time-point visual data. The combination of these two forms a dual spatial and temporal index, enabling precise positioning of the visual data. Multi-resolution pyramid tiles are a data organization format for high-precision visual data in time-domain files, processed by layering and segmenting. The original visual data is generated into multiple "pyramid levels" according to resolution from high to low (the bottom layer consists of micrometer-level high-definition tiles, and the top layer consists of low-definition overview tiles). Each layer is further divided into independent tiles of equal size, and each tile is assigned a unique index, supporting individual retrieval and decoding without loading the entire data.

[0031] Index mapping is a one-to-one correspondence rule between multidimensional index information and multi-resolution pyramid tiles. It binds the combined index of spatial index and time index to the exclusive index of each tile, enabling the system to accurately locate the target tile from the time domain file through dual indexes. It is the key link to realize on-demand data retrieval.

[0032] Specifically, terminal devices (such as tablets, personal computers (PCs), mobile devices, and extended reality (XR) devices (such as augmented reality (AR), mixed reality (MR), and virtual reality (VR) devices) retrieve a specific temporal file (usually with a specific file extension) from local storage or a remote server. This file is not merely a video or image, but a composite data container. It encapsulates massive amounts of visual data with attributes of "multi-viewpoints" (i.e., the same scene captured from different angles) and / or "multi-time points" (i.e., historical data of the scene's evolution over time). The file contains a multi-dimensional index table. This index table records not only the spatial location of the data (latitude and longitude or 3D coordinates) but also its position on the timeline. This index information establishes a mapping relationship, associating the aforementioned spatiotemporal coordinates with the physical storage addresses of the multi-resolution pyramid tiles. This means the system does not need to load the entire large file but can instead "look up" data "on demand" through the index.

[0033] By using temporal files as a unified carrier to encapsulate multi-view / multi-time point visual data, the problems of scattered storage and incompatible parsing of multiple files in traditional demonstrations can be reduced, achieving cross-platform data standardization and reducing the resource consumption of data management and parsing. Multi-dimensional index information is mapped to pyramid tiles, eliminating the need to load the entire high-precision visual data frame; instead, target tiles are retrieved on demand via the index, reducing the transmission, loading, and storage of invalid data from the data layer, significantly reducing system resource consumption. The layered and block-based design of multi-resolution pyramid tiles provides data support for subsequent progressive rendering, structurally solving the problems of lag and long waiting times when loading traditional whole-frame data, thus improving loading efficiency.

[0034] Step 102: Construct a presentation narrative flow based on multidimensional index information. The presentation narrative flow is a spatiotemporal presentation logic constructed based on the multidimensional attributes of the time-domain file. It consists of keyframe sequences and camera motion paths, replacing the linear page sequences of traditional two-dimensional presentations. This enables spatiotemporal presentations from multiple perspectives and at multiple time points, ensuring that the presentation logic matches the multidimensional data attributes of the time-domain file.

[0035] This step is the presentation logic orchestration stage. The presentation narrative flow can include a sequence of keyframes and the camera motion paths of those keyframes. The keyframe sequence is the core node set of the presentation narrative flow. Each keyframe is an important display node in the presentation process, containing camera pose parameters (position, orientation, focal length, etc.) used to define the camera's viewpoint and display state at that node. The keyframes are ordered according to the presentation logic to form a sequence, serving as the start and end points of the camera motion path. The camera motion path is a continuous camera trajectory connecting adjacent keyframes. It can be generated using interpolation algorithms (such as linear, Bézier curve, Catmull-Röhm interpolation, etc.) to allow the camera to smoothly transition from one keyframe to the next. The path can include parameters such as trajectory shape, transition duration, and easing curves, and the trajectory is predictable, serving as the core basis for subsequent predictive loading.

[0036] Specifically, based on the demonstration requirements, several demonstration nodes can be identified as keyframes in the spatial and temporal multidimensional coordinate system of the time-domain file. Camera pose parameters are configured for each keyframe to form a keyframe sequence. Then, an interpolation algorithm is used to generate continuous camera motion paths for adjacent keyframes. The path can be configured with easing curve parameters to control the transition speed, ultimately forming a demonstration narrative flow containing the keyframe sequence and camera motion paths. The path trajectory can be predicted by the system, providing a basis for subsequent prediction and loading.

[0037] Based on multidimensional index information, a directed and predictable narrative logic is built for multidimensional presentations. This can replace the linear page-turning logic of traditional presentations, without modifying the original visual data of the time-domain file. The presentation logic is defined solely by the index, enabling deep matching between the presentation logic and the multi-view / multi-time point attributes of the time-domain file. This reduces the problem of disconnect between traditional presentation logic and multidimensional data, lowers the complexity and resource consumption of presentation arrangement, and improves the narrative coherence of multidimensional presentations. The design of keyframe sequences and camera motion paths makes camera motion trajectories predictable, providing a clear basis for subsequent steps to calculate target areas and retrieve tiles based on path prediction information. This reduces aimless data loading and improves the targeting and loading efficiency of tile retrieval.

[0038] Step 103: During the playback rendering process, the current camera status is obtained in real time, and the target display area within the current view frustum is calculated by combining the predicted information of the camera motion path.

[0039] This step, precise determination of the loading area, is the core of achieving refined data scheduling and is executed during the real-time playback rendering phase of the demonstration. During playback rendering, the current camera's pose state is first acquired in real-time to determine the camera's basic display range. The camera state is the camera's real-time pose information during playback, consistent with the camera pose parameters in the keyframes, including real-time position, orientation, and field of view, used to determine the camera's current frustum and display angle.

[0040] Then, based on the constructed camera motion path, a trajectory prediction algorithm is used to generate predicted information about the camera motion path. The predicted information about the camera motion path is based on the predictability of the camera motion path. It is the information about the relevant areas of the camera's subsequent motion predicted by the algorithm. The core of this information includes the display area corresponding to the next keyframe and the extended view frustum area along the direction of the camera's current path motion. This information is the core input for implementing prediction loading.

[0041] The camera view frustum is a 3D visible area determined by the camera's real-time field of view, position, and orientation. Only the area within the view frustum needs to be loaded and rendered, forming the basis for calculating the target display area and reducing the loading of invalid data outside the view frustum. Based on the current camera view frustum and fused with path prediction information, the target real-world area that needs to be loaded and rendered can be determined through spatial coordinate calculations. The target display area is the core output of this step, referring to the precise spatial area that the system needs to load, decode, and render during the current playback stage. It is calculated by fusing the camera view frustum with path prediction information (optionally fused with the user's region of interest) and serves as the sole basis for subsequently retrieving target tile data, achieving precision in loading the area.

[0042] By acquiring camera status in real time and combining it with the view frustum calculation to determine the basic display range, data processing is performed only on the area within the frustum. This reduces the loading and rendering of invalid data outside the view frustum, significantly reducing unnecessary resource consumption and improving loading efficiency. Integrating predicted camera motion path information to calculate the target area enables predictive loading of the camera's subsequent motion areas. This pre-retrieval of tile data along the path direction reduces waiting time during camera movement, resolving the stuttering issues of traditional on-the-fly loading and achieving seamless loading. Optionally, user interaction operations can be integrated to determine the core target display area, prioritizing the loading of areas of user interest. This allows for refined data scheduling guided by user interests, improving user experience while reducing resource consumption. Precise calculation of the target display area provides a clear basis for the targeted retrieval of subsequent tile data, transforming tile retrieval from blind loading to precise on-demand loading, further improving data scheduling efficiency and reducing system resource consumption.

[0043] Step 104: Based on the target display area and the predicted direction of the camera motion path, retrieve the corresponding target pyramid tile data from the multidimensional index information, and perform progressive decoding and rendering in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

[0044] Specifically, based on both the actual target area and the predicted direction of the camera motion path, the system uses an index mapping relationship to precisely locate and retrieve the corresponding multi-resolution target pyramid tile data from the multi-dimensional index information of the temporal file. The target pyramid tile data is a set of multi-resolution pyramid tiles that precisely match the target display area and the predicted camera motion direction. Retrieved through multi-dimensional index information, it contains only the tiles needed for the current demonstration and contains no invalid data. During retrieval, the priority order is: core target area tiles > predicted direction tiles > ordinary view frustum area tiles, with priority given to loading the core area.

[0045] Then, the retrieved tile data is progressively decoded and rendered in ascending order of resolution. First, the top-level low-resolution tiles are quickly decoded, loaded, and rendered to rapidly form an overall overview of the demonstration scene. Then, the mid- and bottom-level high-resolution tiles are progressively decoded, loaded, and replaced to achieve enhanced overlay of local micro-details. Unlike traditional methods that decode and render the entire dataset at once, this approach first achieves a rapid overall overview before gradually enhancing details.

[0046] Simultaneously, loaded tile data can be cached, establishing a cache eviction mechanism based on access frequency and distance from the current view frustum to release low-priority tiles and ensure efficient utilization of cached resources. Seamless loading refers to a loading experience during the demonstration that achieves rapid overall overview presentation and gradual enhancement of local micro-details, with no stuttering or noticeable waiting when the camera moves / zooms in; it is the final result of the coordinated efforts of all preceding steps.

[0047] Based on the target display area and prediction direction, and combined with index mapping relationships, the system accurately retrieves target tiles, enabling targeted and prioritized retrieval of tile data. This reduces the loading of invalid tiles, further lowers resource consumption, and improves loading efficiency. A progressive decoding and rendering approach, from low to high resolution, is employed. First, a quick overall overview is presented, resolving the issues of long loading times and initial screen wait times associated with traditional high-precision data. Then, fine details are gradually overlaid, balancing loading efficiency and display accuracy to achieve seamless loading from the overall picture to the details. Tile data is cached and an intelligent discard mechanism is established, eliminating the need to reload repeatedly accessed areas. This reduces redundant data transmission and parsing, lowers system resource consumption, and improves loading speed during camera rewind and repeated browsing. The entire loading and rendering process is matched to camera movement paths and user interactions, resulting in no noticeable lag when the camera moves or the user zooms in. This resolves the traditional trade-off between loading efficiency and detail display in multi-dimensional presentations, achieving smooth, seamless browsing of presentation scenes and significantly enhancing the user experience.

[0048] In this embodiment, the temporal file also encapsulates the definition data of interactive units. Interactive units are hidden / interactive information units (i.e., "Easter eggs") embedded in the temporal multidimensional demonstration scene, serving as the core carrier for achieving an immersive demonstration interaction. The definition data of interactive units is structured data specifically used in the temporal file to describe the attributes of the interactive units, including the unique identifier, type, basic resource path, spatiotemporal anchor reference, trigger rule configuration template, etc., of the interactive unit. This data, along with visual data and multidimensional index information, is encapsulated in the temporal file to ensure data integration.

[0049] Based on the interactive unit definition data encapsulated in the temporal domain file, the anchoring of interactive units and the configuration of multimodal triggering rules are completed in the spatiotemporal coordinate system of the presentation narrative flow. By monitoring user interaction operations in real time and performing multimodal combination judgments, the precise triggering of interactive units is achieved. At the same time, a dedicated loading instruction is generated after triggering, prioritizing the retrieval and rendering of high-resolution tile data of the interactive unit area, so that interactive exploration and detail precision are achieved simultaneously, ultimately achieving the dual effects of interactive immersion and precise resource scheduling.

[0050] Specifically, the interactive units are first anchored in the spatiotemporal coordinate system of the presentation narrative flow. Based on the pre-encapsulated interactive unit definition data (including interactive unit type, basic resource information, anchor point positioning benchmark, etc.) in the temporal domain file, there is no need to import external interactive configuration files, ensuring data consistency and encapsulation. Using the spatiotemporal coordinate system of the presentation narrative flow as a benchmark, a unique spatiotemporal anchor point is set for each interactive unit. The anchor point is simultaneously bound to spatial coordinates (corresponding to the specific location of multi-view visual data) and an optional time interval (limiting the interactive unit to be triggered at a specific time stage of the presentation), so that the interactive unit and the spatiotemporal trajectory of the presentation narrative flow are deeply integrated, avoiding the disconnect between the interaction and the main presentation line. The interactive unit is configured with multimodal triggering rules. Multimodal triggering rules are combination logic rules that determine whether user interaction operations can trigger interactive units. The core includes the effective time window of the event and the preset multimodal combination type. Unlike the traditional single click / gaze trigger, it requires at least two interactive operations to form a combination action before a trigger is determined, reducing the probability of false triggering. The core rules can include two main elements: the effective event time window and preset multimodal combination types. Multimodal combinations are rules for determining the combination of at least two interactive operations (not a single trigger), and these rules are associated with the multidimensional index information of the time-domain file, providing a foundation for quickly retrieving associated tiles after subsequent triggers. The effective event time window is a time range for collecting composite interactive signals to avoid false triggers. Starting from the first detection of an interactive signal pointing to the anchor point of the interactive unit, only composite signals within the window participate in the trigger determination; signals outside the window are invalid. The duration can be adjusted according to the scenario.

[0051] During playback, it can be synchronized with camera status monitoring and tile loading to monitor user interaction in real time, such as collecting various types of user interaction signals (e.g., gaze, voice, click, gesture, etc.). The collected signals contain key data such as the spatiotemporal coordinates of the operation, action features, and semantic information, and are matched and calibrated with the spatiotemporal anchor point coordinates of the interaction unit. When any interaction signal is detected pointing to the anchor point area of ​​a certain interaction unit, the effective time window of the event corresponding to that interaction unit is immediately activated, and the centralized monitoring stage of composite interaction signals is entered. Interaction signals that do not point to the anchor point area are directly filtered to reduce the resource consumption of invalid judgments.

[0052] Then, it is determined whether the interactive operation meets the multimodal triggering rules. Within the valid event time window, feature matching and logical judgment are performed on the collected composite interactive signals. The composite interactive signal is a signal combination of at least two interactive operations input by the user within the valid event time window, including the spatiotemporal features, action features, and semantic features of the signal, which is the core basis for multimodal triggering judgment. Only when the signal completely conforms to the preset multimodal combination rules is it determined that the triggering rules are met. If no composite signal conforming to the rules is collected within the window, or only a single interactive signal is collected, the triggering is determined to have failed, the time window is closed, and the normal monitoring state is restored.

[0053] If the interactive operation meets the multimodal triggering rules, the interactive unit is rendered, and a loading instruction for the area where the interactive unit is located is generated. Specifically, according to the preset rendering style (such as a local magnified window, knowledge card, comparison view, measurement tool, etc.) in the interactive unit definition data, the interactive unit content is overlaid at the corresponding spatiotemporal anchor point position in the demonstration scene. During the rendering process, the main demonstration scene is rendered continuously without lag or interruption, and the smoothness of the basic demonstration is not affected. At the same time as the rendering is triggered, a targeted loading instruction is generated in real time. The instruction contains three core pieces of information: the precise spatial coordinates of the area where the interactive unit is located, the associated multidimensional index information identifier, and the high-resolution tile level requirements. The instruction is sent directly to the tile scheduling module, which works in conjunction with the tile loading scheduling to achieve a seamless connection between interaction triggering and instruction generation.

[0054] In response to a loading command, high-resolution tile data associated with the interactive unit is retrieved from the multidimensional index information, with high-resolution pyramid tile data rendered first. The loading command is a directional tile loading control command generated after the interactive unit is triggered. It includes the interactive unit's region coordinates, index identifier, and high-resolution tile level requirements, guiding the tile scheduling module to accurately and preferentially retrieve associated tile data. For example, upon receiving a loading command, based on the spatial coordinates and index identifier in the command, the high-resolution pyramid tile data specific to the interactive unit's region is quickly located from the multidimensional index information of the temporal file. The retrieval process relies on the index mapping relationship of the basic steps, eliminating the need for re-parsed data and ensuring retrieval efficiency. The retrieved high-resolution tile data is set to the highest rendering priority, pausing the replacement of low-resolution tiles in non-core areas, and prioritizing the decoding and rendering of high-resolution tiles in the interactive unit's region to achieve rapid rendering of micron-level details in the interactive unit's region. After the high-resolution tile rendering of the interactive unit's region is completed, the normal tile loading and rendering priority is restored, ensuring the rationality of overall resource scheduling.

[0055] Specifically, an effective event time window is initiated, and composite interaction signals input within this window are monitored. If the composite interaction signal conforms to a preset multimodal combination, it is determined that the multimodal triggering rule is satisfied. The multimodal combination is a preset composite interaction signal matching rule, and the combination type can be expanded according to demonstration needs. For example, a multimodal combination may include a synchronous combination of gaze operation and voice command, or an associated combination of click operation and gesture operation. Specifically, a time window of preset duration can be initiated starting from the first detected interaction signal pointing to the interaction unit anchor point. The window duration can be adjusted according to the demonstration scenario to limit the acquisition time range of composite interaction signals and avoid false triggering. Multiple types of interaction signals are acquired in real time within the window, and the signals are processed for denoising and feature extraction (such as the gaze focus coordinates of gaze signals, semantic parsing of voice signals, and trajectory features of gesture signals).

[0056] In one example, if, within the effective time window of the launch event, the user's gaze is focused on the anchor point area of ​​the interactive unit, and a voice keyword matching a preset semantic library is received simultaneously, then the composite interaction signal is determined to conform to the synchronous combination. In another example, if, within the effective time window of the launch event, a click trigger signal is detected targeting the anchor point area of ​​the interactive unit, and during the duration of the click trigger signal, a gesture operation signal conforming to preset trajectory characteristics is detected, then the composite interaction signal is determined to conform to the associated combination.

[0057] Compared to traditional single-click or single-voice triggers, this application employs a multimodal combination (such as gaze + voice, or click + gesture). This AND logic-based judgment method effectively avoids accidental triggers caused by unconscious actions of users during natural viewing (such as unintentional scanning or background noise), significantly improving the robustness of the interaction. Traditional progressive loading exhibits significant delays when faced with sudden interactions (users need to wait several seconds for the image to become clear after clicking). This application's embodiment, by detecting that the multimodal trigger rules are met, directly retrieves and prioritizes the rendering of high-resolution tiles, achieving a "what you see is what you get" visual effect. This solves the technical challenge of ensuring the clarity of core interactive objects in bandwidth-constrained environments. Furthermore, by anchoring interactive units in a spatiotemporal coordinate system and setting complex trigger rules, the demonstration process is no longer a linear video playback but an immersive experience with exploratory and gamified features, enriching the application scenarios of temporal files.

[0058] In step 102, a spatiotemporal mapping matrix is ​​generated based on the timeline data and spatial coordinate data in the multidimensional index information. The timeline data is structured time-series data extracted from the multidimensional index information of the time-domain file. It is a set of timestamps ordered chronologically, corresponding to the multi-time-point visual data of the time-domain file. It forms the temporal dimension basis for constructing the spatiotemporal mapping matrix and has a unified time calibration benchmark. The spatial coordinate data is a set of three-dimensional spatial coordinates extracted from the multidimensional index information of the time-domain file. It corresponds to the multi-view visual data of the time-domain file, with each coordinate point uniquely identifying a demonstration viewpoint. It forms the spatial dimension basis for constructing the spatiotemporal mapping matrix and has a unified spatial calibration benchmark.

[0059] Then, a two-dimensional spatiotemporal mapping matrix is ​​constructed, with timeline data as the vertical axis (row dimension, labeled with a unique timestamp) and spatial coordinate data as the horizontal axis (column dimension, labeled with a unique spatial coordinate). Each cell of the matrix is ​​a unique spatiotemporal coordinate node, which forms a one-to-one mapping relationship with the corresponding time point and multi-view visual data in the time-domain file. The spatiotemporal mapping matrix is ​​a two-dimensional standardized matrix constructed with timeline data as the vertical axis and spatial coordinate data as the horizontal axis. Each cell of the matrix is ​​a unique spatiotemporal coordinate node, which maps one-to-one with the visual data and tile data index of the time-domain file. It serves as a unified benchmark for achieving spatiotemporal binding of multimedia resources, and its function is to establish a three-dimensional index relationship of time-space-data. Among them, the spatiotemporal coordinate node is the smallest unit in the spatiotemporal mapping matrix, composed of a unique timestamp and spatial coordinates. It is the basic spatiotemporal positioning unit of multimedia resources, ensuring that each type of resource can be accurately bound to a specific time point and specific spatial perspective of the presentation. Each spatiotemporal coordinate node of the spatiotemporal mapping matrix is ​​assigned a value, and core information such as the temporal file visual data index and tile data index associated with the corresponding node is written. At the same time, the matrix is ​​spatiotemporally synchronized and calibrated based on the calibration information of the temporal file to ensure that the spatiotemporal coordinate nodes at different time points and different perspectives are free from offset and conflict, forming a standardized spatiotemporal index benchmark matrix, which provides a unified spatiotemporal positioning basis for subsequent resource binding.

[0060] Then, based on the spatiotemporal mapping matrix, multimedia resources are bound to their corresponding spatiotemporal coordinates to obtain a spatiotemporal associated dataset. This spatiotemporal associated dataset is a structured dataset formed by binding various multimedia resources (native temporal visual data + newly added auxiliary resources) to the spatiotemporal coordinate nodes of the spatiotemporal mapping matrix. It achieves a unified association of time, space, and multimedia resources, serving as the core data foundation for constructing the presentation narrative flow. Specifically, it collects various multimedia resources to be integrated into the presentation, including native multi-view / multi-time-point visual data from temporal files, as well as newly added auxiliary presentation resources (such as audio explanations, text annotations, detailed images, comparison videos, etc.). All resources are standardized in format to ensure they can be parsed and rendered by cross-platform players. Using the spatiotemporal mapping matrix as the sole benchmark, a corresponding target spatiotemporal coordinate node is determined for each type of multimedia resource. Native temporal visual data is directly matched to the associated spatiotemporal coordinate nodes in the matrix, while newly added auxiliary resources are manually / automatically matched to nodes corresponding to specified timestamps and spatial coordinates in the matrix according to presentation requirements. Next, various multimedia resources are deeply bound to their corresponding spatiotemporal coordinate nodes. The binding information includes resource storage path, resource type, rendering priority, and playback parameters (such as audio playback duration and text annotation display duration). All spatiotemporally bound multimedia resources are then structured according to the node order of the spatiotemporal mapping matrix to generate a spatiotemporal associated dataset.

[0061] Finally, based on the preset narrative logic rules, the spatiotemporal related dataset is arranged chronologically and spatially to generate a presentation narrative flow. The narrative logic rules are standardized arrangement rules customized for the presentation scenario, including chronological arrangement rules and spatial layout rules. They can be customized according to different scenarios such as teaching, exhibitions, and press conferences. They are the core basis for guiding the arrangement of the spatiotemporal related dataset, ensuring the logic and coherence of the presentation narrative flow.

[0062] Timing arrangement, based on narrative logic rules, involves sorting and configuring spatiotemporally related datasets along the timeline dimension. It fundamentally determines the presentation's temporal progression, keyframe time nodes, and resource playback sequence, making it a core element in realizing the presentation's temporal narrative. Spatial layout, also based on narrative logic rules, involves laying out and configuring spatiotemporally related datasets along the spatial coordinate dimension. It fundamentally determines the presentation's perspective priority, multi-view switching methods, and resource space overlay rules, making it a core element in realizing the presentation's spatial narrative.

[0063] The key timestamp nodes after temporal arrangement and the key viewpoint nodes after spatial layout are integrated into a sequence of presentation keyframes. Each keyframe contains unique spatiotemporal coordinates, camera pose parameters (automatically generated from spatial layout parameters), and an associated list of multimedia resources. Interpolation algorithms (linear, Bézier curves, etc.) are used to generate continuous camera motion paths for adjacent keyframes. The path parameters are matched with the time intervals of the temporal arrangement and the switching methods of the spatial layout, ultimately forming a complete presentation narrative flow that includes the keyframe sequence, camera motion paths, and multimedia resource playback rules.

[0064] By generating a spatiotemporal mapping matrix, previously discrete and multi-source multimedia resources (such as images, videos, 3D models, and audio) are uniformly mapped into a standardized spatiotemporal coordinate system. This breaks down the barriers between different data formats, enabling the system to manage and call heterogeneous data with a unified logic (i.e., spatiotemporal coordinates). This not only reduces the complexity of data processing but also decouples the data layer from the rendering layer. Subsequent modifications or updates to the narrative flow do not require rebuilding the underlying index, significantly improving the system's flexibility and maintenance efficiency. Timing and spatial layout based on spatiotemporally associated datasets ensure precise synchronization of multimedia resources during presentations. The narrative method based on precise spatiotemporal binding eliminates common data jumps or audio-visual desynchronization issues in traditional presentations, providing users with a coherent, smooth, and highly immersive visual experience. Automated arrangement of spatiotemporally associated datasets using preset narrative logic rules greatly reduces the barrier and workload for manually creating presentation content. Users only need to define the narrative logic (such as "display in chronological order" or "rotate by spatial region"), and the system will automatically generate the corresponding presentation narrative flow based on the spatiotemporal mapping matrix. Furthermore, when new multimedia resources need to be added, they can be automatically integrated into the existing narrative flow simply by binding them to the spatiotemporal mapping matrix, without having to redesign the entire presentation process, which greatly enhances the scalability and reusability of the solution.

[0065] In step 103, the predicted camera pose for the next time step can be calculated first based on the interpolation algorithm of the camera motion path. The next time step is a standardized time unit set for the advance prediction of the camera motion state. It is the length of time the camera will predict forward along the motion path and can be dynamically adapted according to the terminal performance and the demonstration scene to ensure a balance between the advance of the prediction and the accuracy. Based on the transition speed of the camera motion path and the loading performance requirements of the demonstration scene, a standardized next time step is set. This step is the advance prediction time unit of the camera along the motion path to ensure the close fit between the predicted pose and the actual camera motion. The predicted camera pose is the virtual pose state of the camera in the next time step calculated based on the interpolation algorithm. It includes core parameters such as three-dimensional spatial position, viewing angle, and field of view, and is the core benchmark for predicting the view frustum.

[0066] Then, using the intrinsic and extrinsic parameter matrices corresponding to the predicted camera pose, the depth range of the predicted view frustum and its spatial geometric range in the world coordinate system are determined. The intrinsic parameter matrix is ​​a projection transformation matrix determined by the camera hardware parameters. It reflects the mapping relationship between the camera projecting 3D spatial points into 2D image points, including parameters such as focal length, pixel size, and principal point coordinates. It is the fundamental internal parameter for calculating the view frustum and is independent of the camera pose. The extrinsic parameter matrix is ​​a pose transformation matrix determined by the camera's spatial position and orientation. It reflects the camera's position and attitude in the world coordinate system, enabling the mutual conversion between the camera's local coordinate system and the world coordinate system. It is the core external parameter for calibrating the view frustum to the world coordinate system. The predicted view frustum is the camera's forward-predicted visible area calculated based on the predicted camera pose and the intrinsic / extrinsic parameter matrices. It represents the visible area that the camera will cover in the next time step.

[0067] Specifically, by combining the field of view and intrinsic parameter matrix of the predictive camera, the depth range of the predicted view frustum (the distance from the near clipping plane to the far clipping plane) is calculated, defining the effective visible depth of the predictive camera and filtering out invalid spatial regions beyond the depth range, thus reducing the calculation and loading of invalid data at long distances. The extrinsic parameter matrix is ​​used to transform the predicted view frustum from the camera's local coordinate system to the world coordinate system. Combining the depth range and field of view, the complete spatial geometric range of the predicted view frustum in the world coordinate system is calculated (including the vertex of the view frustum, the spatial coordinates of the four sides, and the geometric boundaries of the near / far clipping planes), achieving accurate 3D spatial calibration of the predicted view frustum and ensuring a perfect match between the predicted view frustum range and the spatial coordinates of the actual demonstration scene. The world coordinate system is a unified 3D coordinate system used to describe the position of all spatial objects in the entire demonstration scene. It serves as the common spatial reference for all camera poses, view frustum ranges, and tile data objects in the demonstration scene, ensuring that the view frustum range under different camera states can be subjected to unified spatial calculations.

[0068] Next, the union of the predicted view frustum range and the current view frustum corresponding to the current camera state is used as the target display area. Specifically, based on the real-time pose and intrinsic / extrinsic parameter matrices of the current camera, the current view frustum range (including the spatial geometric range and depth range in the world coordinate system) corresponding to the current camera state is generated according to the same calculation rules as the predicted view frustum, which is the core visible area of ​​the current demonstration. In the world coordinate system, a spatial union operation is performed on the predicted view frustum range and the current view frustum range, and the sum of the coverage areas of the two view frustums is taken as the initial target display area. This area includes both the real-time visible range of the current camera and the predicted visible range that the camera will move to in the next time step, achieving dual coverage of current loading and advance preloading. Slight spatial boundary optimization is performed on the initial target display area to eliminate irregular edge areas after the union of the two view frustums, generating a standardized and continuous final target display area, providing a clear and regular spatial boundary basis for subsequent field-of-view culling.

[0069] Finally, view culling is performed based on the target display area, loading only the target data objects within that area. View culling is a spatial data filtering algorithm in 3D graphics. Using the spatial boundary of the target display area as a benchmark, it removes invalid data objects completely outside the view area, retaining only valid data objects within the view area. Its core function is to reduce the computation and loading of invalid data, improving resource utilization efficiency. For example, using the spatial geometric boundary of the target display area as a criterion, all tile data objects in the temporal file are spatially assigned. Tile data objects completely or partially within the target display area are selected and marked as target data objects to be loaded, while those completely outside the target display area are excluded from loading. The target data objects are multi-resolution pyramid tile data objects that are completely or partially within the target display area after view culling. A second filtering process is then performed on the marked target data objects to be loaded, retaining only multi-resolution pyramid tile data that matches the spatial coordinates and resolution requirements of the target display area, removing redundant data irrelevant to the current resolution and viewpoint, further precisely identifying the tile data that needs to be retrieved. The final filtered target data objects are sorted by spatial location and loading priority to generate a standardized tile data loading list, which is directly passed to the subsequent tile retrieval steps to guide the targeted and accurate loading of data.

[0070] An interpolation algorithm based on camera motion paths calculates the predicted camera pose for the next time step, extending view calculation from real-time to future prediction. It can preload data for the area the camera is about to move to, replacing the traditional passive mode of calculating the view and loading data after the camera moves. This enables predictive, advance loading of tile data, solving problems such as stuttering and blurring caused by untimely data loading during camera movement, providing core technical support for seamless loading. Intrinsic / extrinsic parameter matrices combined with calibrated view frustums achieve precise spatial calibration of the predicted view frustum, ensuring the accuracy of target area calculation. Dual-view frustum union domain extraction improves the continuity of the demonstration. Precise view elimination filters invalid data, significantly reducing resource consumption and loading pressure. Standardized parameter calculation and adaptation improve the universality and cross-platform compatibility of the steps. Dynamically adapted time step design balances prediction accuracy and resource utilization.

[0071] In step 104, a spatiotemporal joint query condition can be constructed based on the geographic spatial range of the target display area and the timestamp interval corresponding to the predicted direction of the camera motion path. The spatiotemporal joint query condition is a combined retrieval condition formed by combining the geographic spatial range of the target display area and the timestamp interval corresponding to the predicted direction of the camera motion. It is used to accurately locate tile data in a multidimensional index, achieving accurate retrieval in both spatial and temporal dimensions. As an example, the corresponding geographic spatial range (spatial boundary in the world coordinate system, geometric range of the view frustum) can be extracted based on the target display area. Simultaneously, the timestamp interval corresponding to the predicted direction of the camera motion path (i.e., the range of the time node the camera is about to reach) can be extracted. The spatial range and timestamp interval are logically combined to construct the spatiotemporal joint query condition, serving as the accurate basis for tile data retrieval.

[0072] Then, using the spatiotemporal joint query conditions, matching is performed within the hierarchical index structure of the multidimensional index information to obtain index nodes that match the spatiotemporal joint query conditions, thus determining the target pyramid tile dataset associated with these index nodes. The hierarchical index structure is a layered index organization method used in multidimensional index information, containing multiple levels of index nodes with a parent-child hierarchical relationship, used to achieve fast retrieval, hierarchical scheduling, and progressive loading of tile data. The query process can start from the top-level parent node, matching child nodes level by level, ultimately locating the index node that fully matches the spatiotemporal joint query conditions. Based on the mapping relationship between index nodes and tile data, the target pyramid tile dataset is determined and obtained. The target pyramid tile dataset is the set of tiles that match the spatiotemporal joint query conditions and are mapped from the index nodes. The target pyramid tile dataset contains multiple data layers divided by resolution levels, and the hierarchical index structure contains a parent-child hierarchical relationship. Parent nodes correspond to large-range, low-resolution tile indexes. Child nodes correspond to small-range, high-resolution tile indexes and are subordinate to the parent nodes. This architecture supports progressive data retrieval from coarse to fine.

[0073] Finally, the target pyramid tile dataset is asynchronously decoded and rendered sequentially according to resolution levels from low to high. A low-resolution overview image is first presented on the canvas, and then the corresponding areas are gradually replaced using high-resolution detail data until the highest resolution level is rendered. Asynchronous decoding is a background decoding method that does not block the main rendering thread, and tiles at multiple resolution levels can be decoded in parallel simultaneously, improving loading efficiency. Progressive rendering renders layers in order of resolution from low to high, first presenting an overview and then gradually replacing it with high-definition details. The entire process is smooth and uninterrupted, achieving seamless loading of the image.

[0074] Based on spatial range and timestamp intervals, joint query conditions are constructed, enabling fast and accurate matching of target tiles in multidimensional indexes. This reduces invalid data retrieval and transmission, significantly improves data retrieval efficiency, and reduces system resource consumption. A hierarchical index structure with parent and child nodes allows the system to perform coarse-to-fine level searches without traversing the entire dataset, significantly accelerating index matching speed, especially suitable for ultra-large-scale multidimensional visual data scenarios. Rendering proceeds sequentially from low to high resolution, first quickly presenting an overall overview and then gradually adding micro-details, eliminating initial screen loading delays and avoiding screen stuttering or flickering, truly achieving seamless loading and smooth presentation of the demonstration scene. Asynchronous decoding of high-resolution tiles does not affect the rapid display of low-resolution images, ensuring uninterrupted playback and improving presentation smoothness. The multi-resolution layered design ensures fast loading, low resource consumption, and ultimately presents the highest precision micro-details, solving the technical problems of slow loading, resource consumption, and low clarity in traditional multidimensional presentations. By combining the predicted direction of camera motion to construct timestamp intervals, predictive preloading is achieved, ensuring that tile data loading always precedes camera movement, further enhancing the continuity and immersion of the presentation.

[0075] In one example, an octree or a K-Dimensional (KD) tree structure can be used to organize the multidimensional index information to establish a hierarchical index structure for spatiotemporal data. An octree is a tree-like data structure used for hierarchical partitioning and indexing of three-dimensional spatial data. Starting from the root node, each three-dimensional spatial node is progressively divided into eight sub-space nodes. It is suitable for constructing spatiotemporal indexes for three-dimensional volumetric and point cloud-like visual data, featuring uniform spatial partitioning and clear hierarchical relationships. A KD tree is a tree-like data structure used for hierarchical indexing of high-dimensional data. By progressively partitioning high-dimensional coordinate data according to specified dimensions, it constructs multi-level index nodes. It is suitable for retrieving high-dimensional coordinate data through spatiotemporal fusion, featuring fast retrieval speed and adaptability to high-dimensional data.

[0076] Specifically, based on the spatial dimensional characteristics of the visual data in the time-domain file (such as 3D spatial point clouds and multi-view planar data), an octree or KD-tree structure can be selected as the organization carrier for multidimensional index information. The octree structure is suitable for the hierarchical division of 3D spatial volume data, while the KD-tree structure is suitable for the rapid retrieval of high-dimensional spatiotemporal coordinate data. Both support hierarchical and regional index management of spatiotemporal data. The multidimensional index information (including spatial index, temporal index, and tile index mapping relationships) of the time-domain file is split according to the principle of spatial dimension as the primary dimension and temporal dimension as the secondary dimension. The core fields of spatiotemporal coordinates are extracted as the key fields of the index structure to ensure accurate matching between the index and the spatiotemporal data. Based on the selected octree / KD-tree, the split multidimensional index information is divided and constructed layer by layer. Starting from the root node (highest level), according to the rule of uniform spatial region division, the large spatial region is divided into multiple sub-spatial regions, each spatial region corresponding to an index node. At the same time, each node is bound to a corresponding timestamp range and tile index basic information, ultimately forming a multi-level, tree-like spatiotemporal data index structure containing root node-parent node-child node, realizing the structured and hierarchical organization of multi-dimensional index information.

[0077] Then, an index is established between the parent node and the low-resolution pyramid tile data, and an index is established between the child nodes and the high-resolution pyramid tiles. First, the multi-resolution pyramid tile data is divided into multiple levels according to resolution from high to low, clarifying the mapping relationship between low-resolution level tiles corresponding to large spatial regions and high-resolution level tiles corresponding to small spatial regions. Next, a unique index is established between the parent node (corresponding to the large spatial region) in the hierarchical index structure and the low-resolution level pyramid tile data. Each parent node is configured with a corresponding low-resolution tile index identifier, storage path, data range, etc., enabling fast indexing of low-resolution tiles by the parent node. Finally, a unique index is established between the child nodes (corresponding to the small spatial regions after the parent node's segmentation) in the hierarchical index structure and the high-resolution level pyramid tile data. Each child node is configured with a corresponding high-resolution tile index identifier, storage path, spatial subdivision range, etc. A parent node can be associated with multiple child nodes, and each child node corresponds to a high-resolution tile within the parent node's spatial region, achieving accurate indexing of high-resolution tiles by the child nodes.

[0078] Next, the spatial regions corresponding to child nodes are configured to completely cover the spatial regions corresponding to parent nodes, establishing a hierarchical inclusion relationship of pyramid tile data in the spatial dimension. This hierarchical inclusion relationship in the spatial dimension refers to the spatial relationship where the spatial region of a parent node is completely covered by the spatial regions of all its child nodes without overlap or omission. Mapped onto pyramid tiles, this means that the large spatial region of a low-resolution tile contains all the small spatial regions of its corresponding high-resolution tile, forming the core spatial foundation for hierarchical tile scheduling. A precise one-to-one mapping is performed between the spatial regions of index nodes and the spatial coverage of pyramid tiles, ensuring that the spatial region of a parent node is completely consistent with the spatial range of its associated low-resolution tile, and the spatial region of a child node is completely consistent with the spatial range of its associated high-resolution tile. All child node spatial regions are configured to completely cover the corresponding parent node spatial region without overlap or omission. This means that the large spatial region of the parent node is evenly divided into multiple small spatial regions of child nodes, and the set of all child node spatial regions is completely equivalent to the parent node's spatial region. This establishes a hierarchical inclusion relationship of pyramid tile data in the spatial dimension, where the large spatial region of a low-resolution tile contains all the small spatial regions of its corresponding high-resolution tile. By combining time indexes, the timestamp ranges of parent and child nodes are configured synchronously to ensure that the timestamp range of child nodes is completely contained within the timestamp range of parent nodes, realizing a hierarchical inclusion relationship in both time and space dimensions, and allowing the index structure to adapt to the needs of time and space joint queries.

[0079] The hierarchical inclusion relationship is used during progressive loading to locate the child node index to be loaded based on the parent node index, enabling hierarchical scheduling of pyramid tile data. This hierarchical scheduling, based on the parent / child node hierarchical inclusion relationship, is a step-by-step, targeted tile data retrieval method, from the parent node to a large range of low-resolution tiles, and then to the child node to a small range of high-resolution tiles. This differs from a full-data retrieval without hierarchy, allowing for fine-grained scheduling of tile data. Specifically, during progressive loading, a large-scale search can be performed using the parent node index to quickly locate the parent node of the area to be loaded, obtaining the associated low-resolution tile data for a rapid overall scene overview. Based on the spatial hierarchical inclusion relationship of parent / child nodes, using the located parent node as a baseline, the corresponding child nodes are retrieved level by level. According to the resolution requirements of the demonstration, the high-resolution tile data associated with the child nodes is retrieved, achieving progressive enhancement of local details. Set up linkage rules between child node retrieval and tile loading. When it is necessary to load high-resolution tiles of a certain area, the child nodes can be located directly through the hierarchical inclusion relationship of the parent node without traversing the entire index structure, thus realizing the directional and hierarchical scheduling of tile data.

[0080] By constructing a hierarchical index structure using octrees / KD-trees, precise associations between nodes and tiles and spatial hierarchical inclusion relationships are established, providing efficient index support for subsequent spatiotemporal joint queries and progressive loading. Compared with traditional non-hierarchical index organization methods, significant improvements are achieved in retrieval efficiency, scheduling accuracy, loading smoothness, and resource utilization.

[0081] Figure 2 This is a schematic diagram illustrating the data structure of a time-domain file and the pyramid tile index provided in an embodiment of this application. For example... Figure 2 As shown in the diagram, the left side of the illustration represents the overall data structure of the temporal domain file's presentation package, comprising three parts: a file header, presentation narrative stream metadata, and multidimensional index information. The file header stores the basic metadata of the temporal domain file, including file version, data verification information, ownership information, encryption information, etc., serving as the entry point for file parsing and validity verification. The presentation narrative stream metadata stores the core configuration information of the presentation narrative stream, including keyframe sequences, camera motion paths, interaction unit anchor points and triggering rules, narrative logic rules, etc., defining the spatiotemporal narrative logic of the presentation. The multidimensional index information, as the core hub for data retrieval, is linked to the multi-resolution pyramid tile index structure on the right, enabling rapid location and hierarchical scheduling of tile data.

[0082] The right side of the diagram shows the hierarchical index structure of the multi-resolution pyramid tiles. For example, it is divided into three layers: L1, L2, and L3 (the number of layers can be expanded). Arrows indicate the hierarchical inclusion relationship from low to high resolution and from large to small area. The L1 layer (top layer) corresponds to the lowest resolution and largest spatial area pyramid tiles, used for quickly loading an overall overview of the demonstration scene. The L2 layer (middle layer) corresponds to medium resolution and medium spatial area pyramid tiles, used to gradually refine scene details. The L3 layer (bottom layer) corresponds to the highest resolution and smallest spatial area segmented pyramid tiles (displayed as a grid structure), used to present local micro-details, achieving progressive loading from the overall to the detailed.

[0083] Multidimensional indexed data is bound to pyramid tiles at each level through index mapping relationships, providing data support for spatiotemporal joint queries and hierarchical scheduling of tile data, enabling seamless loading of demonstration scenarios from overall overview to local micro-details.

[0084] The multi-dimensional presentation method of this application embodiment can be applied to a server. The server is communicatively connected to a presentation terminal and at least one audience terminal. The presentation terminal is operated by the speaker and controls the playback of the presentation narrative stream. It is responsible for broadcasting keyframe parameters to the audience terminals and serves as the narrative reference source for the entire presentation system. The audience terminal is used by the audience and synchronously receives the broadcasts from the presentation terminal. It supports synchronous playback of the main storyline and can initiate local exploration requests to the server based on interactive operations to achieve personalized rendering. This multi-dimensional presentation method also includes the following steps.

[0085] The presentation terminal broadcasts the current playback progress and keyframe parameters of the presentation narrative stream to the audience terminals, enabling the audience terminals to synchronize the playback status of the presentation narrative stream. Keyframe parameters are core configuration parameters used for multi-terminal synchronization, and at least include a keyframe identifier, camera pose (position / orientation), reference view frustum, and timestamp. The reference view frustum is the standard visible view frustum corresponding to the current keyframe on the presentation terminal; it serves as the spatial reference for synchronized playback on all audience terminals and is also the reference coordinate system for calculating the local extended region of the audience.

[0086] While playing the presentation narrative stream (composed of a sequence of keyframes and camera motion paths), the presentation terminal collects the current playback progress (such as timestamps and current keyframe identifiers) and keyframe parameters in real time. The keyframe parameters include at least: camera pose parameters (position, orientation, focal length / field of view), reference frustum range (the current standard visible area in the world coordinate system), and the corresponding timestamp interval. The presentation terminal encapsulates the broadcast command containing the playback progress and keyframe parameters via a network communication protocol and broadcasts it to all connected audience terminals. Upon receiving the broadcast command, the audience terminals parse the keyframe parameters, synchronously load the low-resolution pyramid tile data corresponding to the keyframe, and jump to a reference viewpoint completely consistent with the presentation terminal, completing the multi-terminal synchronized playback of the presentation narrative stream.

[0087] During synchronized playback, the audience terminal detects user interaction (such as finger swiping, mouse dragging, gamepad control, etc.) and triggers an interaction. When the detected interaction intent is "local exploration," the camera yaw angle parameter is extracted. This camera yaw angle parameter characterizes the horizontal viewing angle offset of the audience camera relative to the reference camera (the reference viewpoint center of the presentation terminal), defining the direction and range of the audience's exploration. The audience terminal assembles a local exploration request containing the camera yaw angle parameter and sends it to the server, which receives the request. The local exploration request includes the camera yaw angle parameter triggered by the audience terminal's interaction. The camera yaw angle parameter is the horizontal rotation angle (yaw angle) of the audience terminal's camera relative to the reference camera (the presentation terminal's viewpoint), used to quantify the audience's exploration direction and serving as the core geometric input for calculating the local expansion area.

[0088] Then, based on the camera yaw angle parameters and the reference view frustum corresponding to the current playback progress, the server calculates the locally extended display area of ​​the audience terminal at the current narrative node. The locally extended display area is the personalized visual area that the audience terminal explores beyond the range of the reference view frustum through interactive exploration. This area belongs only to that audience member and does not affect the view of the speaker terminal or other audience members. Upon receiving an exploration request, the server obtains the reference view frustum corresponding to the current playback progress synchronized with the speaker terminal (i.e., the standard field of view currently focused on by the speaker). Using the reference view frustum as a spatial reference, spatial coordinates are calculated based on the received camera yaw angle parameters. Through a rotation matrix or coordinate transformation algorithm, the audience's yaw angle perspective is mapped to the world coordinate system, calculating the locally extended display area corresponding to the audience's exploration perspective at the current narrative node.

[0089] Next, the incremental multidimensional index information corresponding to the locally expanded display area is fed back to the viewer terminal, instructing the viewer terminal to retrieve the corresponding tile data for independent rendering based on the multidimensional index information. The incremental multidimensional index information is index data extracted by the server only for the locally expanded display area. Compared to the full index, its data volume is extremely small, containing only the tile index mapping relationship of the expanded area, used for quickly locating and loading tiles in the expanded area. Based on the calculated locally expanded display area, the server quickly retrieves and extracts the corresponding incremental multidimensional index information from the multidimensional index information in the temporal domain file. The incremental multidimensional index information only contains the index data of the locally expanded area and does not contain the baseline frustum index data that the viewer terminal has already synchronously loaded via the mainline, thus minimizing data transmission. The server feeds back the extracted incremental multidimensional index information point-to-point to the viewer terminal that sent the exploration request. The viewer terminal receives the incremental multidimensional index information, retrieves the corresponding pyramid tile data (usually high-resolution detail tiles) from the temporal domain file based on this index information, and renders it independently on the local canvas.

[0090] Figure 3 This is a timing diagram for the synchronization control of a presentation terminal and an audience terminal provided in an embodiment of this application. For example... Figure 3 As shown in the attached diagram, this multi-dimensional, dual-track presentation process involves collaboration between the speaker terminal, server, and audience terminal, enabling a synchronized mechanism for the speaker's main control and the audience's personalized exploration. This synchronized control process may include the following steps.

[0091] S1. The presentation terminal broadcasts the playback command of keyframe ID=K1 to the audience terminal.

[0092] S2. The viewer terminal parses K1 and initiates a request for the tile data corresponding to K1.

[0093] S3: The speaker terminal forwards the request and requests tile data at the K1 level from the server.

[0094] S4. The server pushes / downloads the low-resolution tile data corresponding to K1 to the viewer's terminal.

[0095] S5. The viewer's terminal loads the tile, jumps to the K1 perspective, displays an overall scene overview, and completes the main storyline synchronization.

[0096] S6. The audience terminal initiates a partial exploration request, which is forwarded to the server via the presentation terminal.

[0097] S7. The server pushes the incremental multidimensional index information corresponding to the exploration area to the viewer's terminal.

[0098] S8: The viewer terminal retrieves tiles based on incremental indexing and renders and displays the locally expanded exploration area.

[0099] S9. The presentation terminal advances the demonstration and broadcasts the instruction for the new keyframe ID=K2 to the audience terminals.

[0100] S10: The audience terminal smoothly transitions from the exploration state to the main perspective corresponding to K2, completing the demonstration transition.

[0101] In traditional solutions, viewers either passively follow the speaker (without seeing details) or need to reload the entire scene (disrupting the main narrative). This application's embodiment ensures narrative consistency through broadcast synchronization of the main narrative, while allowing viewers to freely explore details of interest without affecting the main narrative through local exploration requests. This satisfies both the speaker's narrative control needs and the viewer's desire for detail, resolving the conflict between synchronization and exploration. Instead of pushing incremental multi-dimensional index information, the entire scene data is pushed. Since the viewer's terminal has already loaded most of the basic data synchronously through the main narrative, the server only needs to push incremental indexes for the exploration area, minimizing data transmission. The viewer's terminal only needs to load a small number of tiles corresponding to the incremental index, without needing to re-decode the entire scene, significantly reducing rendering pressure. This design ensures smooth parallel exploration for multiple viewers even on mobile devices and in weak network environments. Using the speaker's reference view frustum as a unified coordinate system, combined with camera yaw angle calculations, ensures that the area explored by the viewer perfectly matches the world coordinate system and the presentation scene, reducing scene misalignment and tile loading chaos caused by different perspectives from multiple devices. Every deviation in the audience's exploration is precisely mapped to the corresponding location in the demonstration scene, enhancing the realism of the exploration experience. Local exploration requests and feedback are point-to-point interactions between the server and a single audience terminal, without broadcasting to the entire network, thus not affecting the main broadcast efficiency of the presentation terminal or interfering with other audience members' viewing. Independent rendering means that each audience terminal's rendering task runs independently, avoiding the domino effect of a single audience member's lag causing a global lag, and improving the stability and concurrency of the entire demonstration system. All data interactions (synchronization, requests, retrieval, rendering) in this embodiment are based on the core technology system of temporal files, multidimensional indexes, and pyramid tiles, without introducing external dependencies and fully reusing existing index structures, resulting in a complete closed-loop technical solution.

[0102] Figure 4 This is a schematic diagram of the structure of a multi-dimensional demonstration device based on time-domain technology provided in an embodiment of this application. Figure 4 As shown, the multidimensional demonstration device 400 based on time-domain technology may include an acquisition module 401, a construction module 402, a calculation module 403, and a rendering module 404.

[0103] The acquisition module 401 is used to acquire a time-domain file, which contains visual data from multiple perspectives and / or multiple time points, and encapsulates multi-dimensional index information containing spatial and temporal indices. The multi-dimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles.

[0104] The construction module 402 is used to construct a presentation narrative stream based on multidimensional index information. The presentation narrative stream includes the keyframe sequence of the presentation and the camera motion path of the keyframes.

[0105] The calculation module 403 is used to obtain the current camera status in real time during the playback rendering process, and calculate the target display area within the current view frustum by combining the prediction information of the camera motion path.

[0106] The rendering module 404 is used to retrieve the corresponding target pyramid tile data from the multidimensional index information based on the predicted direction of the target display area and the camera motion path, and to perform progressive decoding and rendering in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

[0107] The acquisition module 401, construction module 402, calculation module 403 and rendering module 404 can be used to execute steps 101-104 in the embodiments of the multidimensional demonstration method based on time domain technology. For the specific implementation of these modules and more details, please refer to the corresponding method section, which will not be elaborated here.

[0108] This application also provides a computer-readable storage medium storing a program that can be loaded and executed by a processor, which is any of the time-domain-based multidimensional demonstration methods in this application.

[0109] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0110] The above examples illustrate this application only to aid understanding and are not intended to limit its scope. Those skilled in the art to which this application pertains can make various simple deductions, modifications, or substitutions based on the ideas presented.

Claims

1. A multidimensional demonstration method based on time-domain techniques, characterized in that, include: Obtain a time-domain file containing visual data from multiple viewpoints and / or multiple time points, and encapsulate multi-dimensional index information containing spatial and temporal indexes, wherein the multi-dimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles; Based on the time axis data and spatial coordinate data in the multidimensional index information, a spatiotemporal mapping matrix is ​​generated; Based on the spatiotemporal mapping matrix, multimedia resources are bound to corresponding spatiotemporal coordinates to obtain a spatiotemporal associated dataset; Based on preset narrative logic rules, the spatiotemporal related dataset is arranged temporally and spatially to generate a demonstration narrative stream, which includes a sequence of keyframes for the demonstration and the camera motion paths of the keyframes. During the playback rendering process, the current camera status is acquired in real time, and the target display area within the current view frustum is calculated by combining the predicted information of the camera motion path. Based on the predicted direction of the target display area and the camera motion path, the corresponding target pyramid tile data is retrieved from the multidimensional index information, and progressively decoded and rendered in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

2. The multidimensional demonstration method based on time-domain technology according to claim 1, characterized in that, The time-domain file also encapsulates the definition data of the interactive units, and the multi-dimensional demonstration method further includes: The interactive unit is anchored in the spatiotemporal coordinate system of the demonstration narrative flow, and the interactive unit is configured with multimodal triggering rules; During playback, the system monitors user interactions in real time and determines whether the interactions satisfy the multimodal triggering rules. If the interactive operation satisfies the multimodal triggering rule, the interactive unit is triggered to be presented, and a loading instruction for the area where the interactive unit is located is generated; In response to the loading command, the high-resolution tile data associated with the interactive unit is retrieved from the multidimensional index information, and the high-resolution pyramid tile data is rendered first.

3. The multidimensional demonstration method based on time-domain technology according to claim 2, characterized in that, The step of determining whether the interaction operation satisfies the multimodal triggering rule includes: Start the event validity time window and monitor the composite interactive signals input within the event validity time window; If the composite interaction signal conforms to the preset multimodal combination, it is determined that the multimodal triggering rule is satisfied. The multimodal combination includes a synchronous combination of gaze operation and voice command or an associated combination of click operation and gesture operation. Wherein, the step of determining that the multimodal triggering rule is satisfied if the composite interaction signal conforms to a preset multimodal combination includes: If, within the effective time window of the launch event, the user's gaze is detected focusing on the anchor point area of ​​the interaction unit, and a voice keyword matching a preset semantic library is received simultaneously, then the composite interaction signal is determined to conform to the synchronous combination; or If a click trigger signal is detected for the anchor point area of ​​the interaction unit within the effective time window of the launch event, and a gesture operation signal conforming to the preset trajectory characteristics is detected during the duration of the click trigger signal, then the composite interaction signal is determined to conform to the associated combination.

4. The multidimensional demonstration method based on time-domain technology according to claim 1, characterized in that, During the playback rendering process, the current camera state is acquired in real time, and the target display area within the current view frustum is calculated by combining the predicted information of the camera motion path. This includes: Based on the interpolation algorithm of the camera motion path, calculate the predicted camera pose for the next time step; Using the intrinsic and extrinsic parameter matrices corresponding to the predicted camera pose, the depth range of the predicted view frustum and the spatial geometric range of the depth range in the world coordinate system are determined. The union of the predicted view frustum range and the current view frustum corresponding to the current camera state is taken as the target display area; View culling is performed based on the target display area, and only the target data object of the target display area is loaded.

5. The multidimensional demonstration method based on time-domain technology according to claim 1, characterized in that, The step of retrieving the corresponding target pyramid tile data from the multidimensional index information based on the predicted direction of the target display area and the camera motion path, and progressively decoding and rendering it in order of increasing resolution, includes: Based on the geographic spatial range of the target display area, and combined with the timestamp interval corresponding to the predicted direction of the camera motion path, a spatiotemporal joint query condition is constructed. The spatiotemporal joint query conditions are used to match the hierarchical index structure of the multidimensional index information to obtain the index node that matches the spatiotemporal joint query conditions, and to determine the target pyramid tile dataset associated with the index node. The target pyramid tile dataset contains multiple data layers divided according to resolution levels, and the hierarchical index structure contains the hierarchical inclusion relationship between parent nodes and child nodes. The target pyramid tile dataset is asynchronously decoded and rendered sequentially according to the resolution levels from low to high. A low-resolution overview image is first presented on the canvas, and then the corresponding areas are gradually replaced using high-resolution detail data until the rendering of the highest resolution level is completed.

6. The multidimensional demonstration method based on time-domain technology according to claim 5, characterized in that, Also includes: The multidimensional index information is organized using an octree or KD tree structure to establish a hierarchical index structure for spatiotemporal data. The parent node is associated with the low-resolution pyramid tile data through an index, and the child node is associated with the high-resolution pyramid tile data through an index. Configure the spatial region corresponding to the child node to completely cover the spatial region corresponding to the parent node, so as to establish the hierarchical inclusion relationship of the pyramid tile data in the spatial dimension; The hierarchical inclusion relationship is used to locate the child node index to be loaded based on the parent node index during the progressive loading process, thereby realizing the hierarchical scheduling of the pyramid tile data.

7. The multidimensional demonstration method based on time-domain technology according to claim 1, characterized in that, The multidimensional presentation method is applied to a server, which is communicatively connected to a speaking terminal and at least one audience terminal. The multidimensional presentation method further includes: The presentation terminal broadcasts the current playback progress and keyframe parameters of the presentation narrative stream to the audience terminal, so that the audience terminal can synchronize the playback status of the presentation narrative stream. The server receives a local exploration request sent by the viewer terminal, the local exploration request including camera yaw angle parameters triggered by the viewer terminal's interaction; Based on the camera yaw angle parameters and the reference frustum corresponding to the current playback progress, the local extended display area of ​​the viewer terminal at the current narrative node is calculated. The incremental multidimensional index information corresponding to the local extended display area is fed back to the viewer terminal, so as to instruct the viewer terminal to retrieve the corresponding tile data based on the multidimensional index information for independent rendering.

8. A multidimensional demonstration device based on time-domain technology, characterized in that, include: The acquisition module is used to acquire a time-domain file, which contains visual data from multiple perspectives and / or multiple time points, and encapsulates multi-dimensional index information containing spatial and temporal indexes. The multi-dimensional index information is associated with the index mapping relationship of multi-resolution pyramid tiles. The construction module is used to generate a spatiotemporal mapping matrix based on the time axis data and spatial coordinate data in the multidimensional index information, bind multimedia resources with corresponding spatiotemporal coordinates according to the spatiotemporal mapping matrix to obtain a spatiotemporal associated dataset, and perform temporal arrangement and spatial layout on the spatiotemporal associated dataset based on preset narrative logic rules to generate a demonstration narrative stream. The demonstration narrative stream includes a sequence of key frames of the demonstration and the camera motion path of the key frames. The calculation module is used to obtain the current camera status in real time during the playback rendering process, and calculate the target display area within the current view frustum by combining the prediction information of the camera motion path. The rendering module is used to retrieve the corresponding target pyramid tile data from the multidimensional index information based on the predicted direction of the target display area and the camera motion path, and to perform progressive decoding and rendering in order of resolution from low to high, so as to achieve seamless loading of the demonstration scene from overall overview to local micro-details.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that can be loaded by a processor and executed as described in any one of claims 1 to 7, the multidimensional demonstration method based on time-domain techniques.

Images