An immersive interaction method and device for appreciating an object

Abstract

The application discloses an immersion interactive method and device for appreciating artifacts, and relates to the technical field of human-computer interaction. The method comprises the following steps: in an immersive digital experience cabin, the three-dimensional coordinates of the user's double pupils and the hand skeleton node coordinates are collected, the gesture instruction is analyzed, and the three-dimensional space viewpoint parameter set of the next moment is predicted; in a virtual engine scene, the UV texture coordinates are obtained by ray intersection, the three-dimensional region of interest is demarcated, the candidate tile set and the tile topology feature matrix are retrieved and generated; based on the pre-constructed fitness function, the improved tornado optimization algorithm is used for solving, and the global optimal tile loading sequence is output; the naked-eye light field rendering and the three-dimensional space multi-modal superposition are performed on the global optimal tile loading sequence, and the fusion picture of the artifacts and the suspended interpretation is presented on the naked-eye 3D cabin screen. The application solves the problems of viewpoint prediction lag, texture tile loading leading to video memory overflow or picture damage, and lack of spatial multi-modal fusion in the prior art.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an immersive interactive method and device for appreciating objects. Background Technology

[0002] With the development of digital museums and 3D digitization technology, immersive appreciation of precious artifacts (such as bronzes, ceramics, and jade) using glasses-free 3D devices has become a trend. Immersive digital experience cabins, integrating multi-channel high-definition projection / display, glasses-free 3D, spatial audio, haptic interaction, and real-time rendering technologies in enclosed or semi-enclosed spaces, have emerged to provide users with a comprehensive and highly immersive digital experience. The advent of glasses-free 3D technology and light field display technology allows viewers to obtain a three-dimensional and realistic visual experience in immersive digital experience cabins without wearing heavy VR headsets.

[0003] However, existing immersive artifact appreciation technologies have the following significant technical shortcomings in practical applications: 1) Interaction Delay and Viewpoint Lag: When users zoom in and out using gestures or change their viewpoint by moving their heads inside the cabin, a synchronous mechanism of "capture first, then respond" is usually used. Because the rendering of high-precision 3D models and the loading of ultra-high-resolution textures consume a lot of time, there is a noticeable lag in screen transitions, which can easily cause users to feel dizzy.

[0004] 2) Rigid texture loading strategy: Existing technologies mostly use quadtree or octree detail level models based on fixed levels. When users quickly focus on a small area (such as the inscription or glaze crack of an object), they cannot dynamically plan the best texture tile loading order according to network bandwidth and GPU memory status, which often leads to "mosaic" on the screen or program crashes due to memory overflow.

[0005] 3) Lack of multimodal spatial fusion: Existing naked-eye 3D displays mostly stay at the "seeing" level, failing to anchor the microscopic features of the object's surface with the structured interpretation data in the knowledge graph in three-dimensional space, resulting in a significant reduction in the immersive interactive experience of "what you see is what you know".

[0006] Therefore, there is an urgent need for an immersive interactive method for appreciating objects that can take into account low-latency viewpoint prediction, intelligent tile scheduling that adapts to video memory and bandwidth, and fusion of three-dimensional spatial multimodal modes. Summary of the Invention

[0007] This invention provides an immersive interactive method and device for appreciating objects. This invention solves the problems of viewpoint prediction lag, easy memory overflow or image damage caused by texture tile loading, and lack of spatial multimodal fusion in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide an immersive interactive method for appreciating artifacts, the method comprising: In the immersive digital experience cabin, the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bones are collected, the gesture commands are analyzed, and the three-dimensional spatial viewpoint parameter set for the next moment is predicted and generated. Based on the three-dimensional spatial viewpoint parameter set, ray intersection is performed in the virtual engine scene to obtain UV texture coordinates, three-dimensional regions of interest are delineated, and candidate tile sets and tile topological feature matrices are retrieved and generated. Based on the candidate tile set and tile topology feature matrix, and using a pre-constructed fitness function, an improved tornado optimization algorithm is employed to solve the problem and output the globally optimal tile loading sequence. The globally optimal tile loading sequence is rendered using naked-eye light field rendering and superimposed with three-dimensional spatial multimodal rendering, and the fused image of objects and suspended interpretation is presented on the naked-eye 3D cabin screen of the immersive digital experience cabin.

[0009] The technical solution provided in this application has at least the following beneficial effects: A Kalman filter based on a uniform motion model combined with real-time perturbation is introduced to predict the binocular center, gaze direction, and pinch scaling gain in a unified priori manner. This allows for accurate prediction of the viewpoint parameter set for the next frame in the current frame, enabling subsequent ray intersection and texture loading to be performed one time step ahead. This fundamentally eliminates the visual dragging and dizziness in naked-eye 3D environments, as well as the visual lag caused by head movements and gestures. For high-precision object textures, a fitness function is constructed with three major constraints: visual area coverage, network bandwidth load, and memory fluctuation. By introducing Logistic chaotic mapping initialization and an improved Tornado optimization algorithm that integrates PSO mechanism, the globally optimal tile loading sequence with absolute safety and highest image quality is calculated in an extremely short time, eliminating frame drops and memory overflow crashes. In the light field rendering stage, not only is a true naked-eye 3D effect based on micro-viewpoint sub-image synthesis achieved, but the object's 3D knowledge graph can also be directly queried through the coordinates of the ray hit point. The structured explanation data is instantiated into a UI floating panel and accurately superimposed on the 3D hit point position, achieving a deep fusion appreciation experience of "what you see is what you know".

[0010] In one alternative implementation, within an immersive digital experience cabin, the user's binocular 3D coordinates and hand skeletal node coordinates are collected. Gesture commands are parsed, and a set of 3D spatial viewpoint parameters for the next moment is predicted and generated, including: In the immersive digital experience cabin, a physical world coordinate system is established with the center of the bottom surface of the naked-eye 3D cabin as the origin; An eye-tracking module is used to collect the three-dimensional coordinates of the user's pupils, and a motion-sensing module is used to collect the coordinates of the user's hand skeletal nodes. Extract the coordinates of the thumb tip and index finger tip from the hand bone node coordinates, calculate the Euclidean distance and pinching rate between them, perform gesture semantic parsing, and if the air focus zoom command is triggered, calculate the zoom gain coefficient of the virtual camera and proceed to the next step. If both the Euclidean distance and the pinch rate are greater than the corresponding threshold, the remote focus scaling command is triggered, and the scaling gain coefficient of the virtual camera is calculated based on the absolute value of the pinch rate. Based on the three-dimensional coordinates of the two pupils, calculate the coordinates of the center point of the two pupils, and construct the gaze direction vector based on the coordinates of the center point of the two pupils, the coordinates of the thumb tip, and the coordinates of the index finger tip; Kalman filtering is used to dynamically predict the three-dimensional spatial viewpoint parameter vector, which consists of the coordinates of the center points of the two pupils, the gaze direction vector, and the scaling gain coefficient, to obtain the three-dimensional spatial viewpoint parameter set for the next moment. The three-dimensional spatial viewpoint parameter set includes the predicted coordinates of the center points of the two pupils, the predicted gaze direction vector, and the predicted scaling gain coefficient.

[0011] In one optional implementation, a Kalman filter is used to dynamically predict the three-dimensional spatial viewpoint parameter vector, which consists of the coordinates of the dual pupil center points, the gaze direction vector, and the scaling gain coefficient, to obtain the three-dimensional spatial viewpoint parameter set for the next time step, including: By integrating the current coordinates of the center points of the two pupils, the gaze direction vector, and the scaling gain coefficient, a three-dimensional spatial viewpoint parameter vector is obtained. Within a unit time step, assuming that the user's head movement, gaze rotation, and gesture zoom are all uniform motions, construct a state transition matrix based on the uniform motion model; Based on the uniform motion model, the control input vector and control matrix are defined. Based on the state transition matrix, control input vector, and control matrix, execute the first prediction equation to calculate the prior state estimate of the three-dimensional spatial viewpoint parameter vector at the next time step and the corresponding prior error covariance matrix. When the next moment arrives, the raw measurement values ​​of the eye-tracking module and the motion sensing module are collected and packaged into the observation vector for the next moment; Construct the observation matrix and measurement noise covariance matrix, and calculate the Kalman gain; Based on the Kalman gain, the prior state estimate and the observation vector are weighted and fused to obtain the optimal posterior estimate for the next time step. Update the error confidence matrix to prepare for the next iteration; The three-dimensional spatial viewpoint parameter set for the next time step is extracted from the optimal posterior estimate.

[0012] In one optional implementation, based on the three-dimensional spatial viewpoint parameter set, ray intersection is performed in the virtual engine scene to obtain UV texture coordinates, a three-dimensional region of interest is defined, and a candidate tile set and a tile topological feature matrix are retrieved and generated, including: In the virtual engine scene, a three-dimensional ray is emitted along the direction of the predicted line of sight vector, with the coordinates of the predicted dual pupil center point in the three-dimensional spatial viewpoint parameter set as the ray origin. Using a GPU-accelerated hierarchical bounding box traversal algorithm, the nearest intersection point of the 3D ray and the 3D mesh of the object in the naked-eye 3D cabin is calculated, and the 3D coordinates of the nearest intersection point and the index of the hit triangle face in the local coordinate system of the model are obtained. Read the data of the three vertices of the triangle corresponding to the index of the hit triangle and their bound texture coordinates. Use the 3D space volume ratio to calculate the weight of the 3D coordinates of the intersection point relative to the centroid coordinates of the triangle. Then, use the interpolation formula to back-calculate the UV texture coordinates of the hit point in the 2D texture space. Based on the predicted scaling gain coefficients in the three-dimensional spatial viewpoint parameter set, the physical size of a single screen pixel in UV space is calculated at the naked-eye 3D cabin screen resolution, and the side length of the rectangular region corresponding to the three-dimensional region of interest is calculated. Using the UV texture coordinates as the geometric center, the rectangular region corresponding to the 3D region of interest is extended in the positive and negative directions of the horizontal and vertical axes of the UV plane to define a 3D region of interest with a fixed aspect ratio. Calculate the target texture layer level based on the predicted scaling gain coefficients in the three-dimensional spatial viewpoint parameter set; Traverse the tiles in the target layer and adjacent layers of the tile server, extract several tiles that spatially overlap with the 3D region of interest, and form a candidate tile set; Extract the center UV coordinates, tile data volume, and level in the pyramid for each tile in the candidate tile set, and construct the tile topology feature matrix.

[0013] In one optional implementation, based on the candidate tile set and the tile topology feature matrix, and using a pre-constructed fitness function, an improved tornado optimization algorithm is employed to solve for the globally optimal tile loading sequence, including: The tile loading priority vector of all candidate tiles in the candidate tile set is encoded into the position vector of an individual in the improved tornado optimization algorithm, and combined with the tile topology feature matrix, the corresponding fitness function is constructed. The chaotic sequence is generated using Logistic mapping, and then mapped to the integer range of individuals in the improved tornado optimization algorithm through carrier mapping and deduplication, generating several random integer permutations as the initial individuals to form the initial population. Using the fitness function, the fitness value of each initial individual in the initial population is calculated, and based on the fitness value, the optimal position of each initial individual and the global optimal individual in the initial population are determined. A convergence factor and a PSO mechanism are introduced, and the positions of the initial population or the population updated in the previous iteration are updated based on the individual optimal position and the global optimal individual to obtain the updated population. Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; Repeatedly update the position of the population. When the current iteration reaches the maximum iteration count or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual at the current iteration count. The position vector of the globally optimal individual is decoded to obtain the globally optimal tile loading sequence.

[0014] In one alternative implementation, the fitness function is formulated as follows: In the formula, For the first i The fitness value of an individual; For the first i The total number of tiles in the candidate tile loading priority vector corresponding to an individual; i For individual indicators; j For tile indication; The tile topological feature matrix is ​​the first... i The first individual j The tile data volume, which is directly derived from the tile data volume in the 3rd column of the tile topology feature matrix; The estimated network bandwidth between the eye-tracking module and the motion-sensing module and the local edge rendering server within the immersive digital experience cabin; This refers to the single-frame rendering time interval of the rendering engine; This represents the current available remaining video memory capacity of the GPU; The region of interest is a three-dimensional region. This is the area calculation function; For the first i The first individual j The two-dimensional geometric rectangular area occupied by the tile on the UV plane; This is the union operation term for the two-dimensional geometric rectangular region occupied by the tile on the UV plane; The symbol for the intersection of sets; This refers to the fitness weighting coefficient.

[0015] In one optional implementation, a convergence factor and a PSO mechanism are introduced, and the initial population or the population updated in the previous iteration is updated based on the individual optimal position and the globally optimal individual, resulting in an updated population, including: The initial population or the population updated in the previous iteration is divided into the tornado population with the highest fitness value, the thunderstorm population with the middle fitness value, and the storm population with the lowest fitness value. By introducing a convergence factor and a PSO mechanism, based on the individual optimal position and the global optimal individual, the tornado population is finely searched around the global optimal individual to obtain an updated first population. By introducing convergence factor, PSO mechanism and tornado rotation term, based on individual optimal position and global optimal individual, the thunderstorm population explores and develops around the global optimal individual, resulting in a newer second population; By introducing a convergence factor, a PSO mechanism, and a stochastic traction term, the storm population is made to explore the world around the globally optimal individual based on the individual's optimal position and the globally optimal individual, thus obtaining an updated third population. By integrating the updated first population, the updated second population, and the updated third population, an updated population is obtained.

[0016] In one optional implementation, the globally optimal tile loading sequence is rendered using naked-eye light field rendering and superimposed with three-dimensional spatial multimodal rendering, and the fused image of the object and its suspended interpretation is presented on the naked-eye 3D screen of the immersive digital experience cabin, including: Based on the optimal fitness value corresponding to the globally optimal tile loading sequence, if the optimal fitness value is greater than the preset safety threshold, it is determined that there is a risk of frame drop, the prediction scaling gain coefficient in the three-dimensional spatial viewpoint parameter set is reduced, and the three-dimensional region of interest delineation step is returned. The globally optimal tile loading sequence is parsed, and network fetch requests are sent to the tile server in order of optimal tile loading priority. Based on the network fetch request, the GPU memory controller reads the tile data volume of each tile in the globally optimal tile loading sequence, performs pre-allocation and page locking of continuous virtual memory in the memory pool, and completes the tile loading. During the GPU pixel shading stage, the 3D region of interest is read, and the pixel UV texture coordinates of each currently rendered pixel on the object's 3D mesh are extracted. Calculate the tile row and column index that the pixel should currently hit based on the pixel UV texture coordinates and the target texture level corresponding to the predicted scaling gain coefficient; If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has been loaded, then the pixel is determined to belong to the high-optimal loading region, and full-resolution high dynamic range PBR material sampling is performed on the pixel. If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has not been fully loaded, then it is determined that the high-definition tile corresponding to the pixel is not yet ready, and the low-level blur map resident in the video memory is directly sampled for the pixel. If the tile row and column index does not exist in the global optimal tile loading sequence, then the pixel is determined not to belong to the core focus area, and the low-level blur map resident in the video memory is directly sampled for the pixel. Traverse all pixels of the object's 3D mesh, repeat the GPU pixel shading process, and combine the predicted dual pupil center point coordinates from the 3D spatial viewpoint parameter set to generate several micro-viewpoint sub-images around the predicted dual pupil center point coordinates using a light field synthesizer. Several micro-perspective sub-images are synthesized to obtain a micro-perspective composite image of the object, and the micro-perspective composite image is output to the naked-eye 3D cabin screen of the immersive digital experience cabin to complete naked-eye light field rendering. Extract the 3D coordinates of the nearest intersection point, and use these 3D coordinates as the query key to input them into a pre-built 3D knowledge graph database of artifacts. If a matching entity exists for the query key, then retrieve the structured explanation data corresponding to that matching entity. In the virtual engine scene, the corresponding UI floating panel is instantiated, and the structured explanation data is rendered onto the UI floating panel to obtain the floating interpretation panel. The floating interpretation panel is then superimposed onto the micro-view synthesized image on the naked-eye 3D cabin screen to complete the three-dimensional space multimodal superposition.

[0017] In one alternative implementation, the structured explanation data includes the explanatory text string, highlighted material attribute parameters, and the explanation voice data stream; The explanatory text strings in the structured explanatory data are drawn onto the preset layout area of ​​the UI floating panel using a text rendering component; Extract the self-illuminating color and transparency coefficient of the highlighted material attribute parameters from the structured explanation data, and apply the self-illuminating color and transparency coefficient to the background mask and / or the underlying 3D surface of the UI floating panel to generate a floating interpretation panel with visual focus guidance effect. Register the three-dimensional spatial coordinates of the floating interpretation panel as a spatial audio emission source anchor point, call the spatial audio module, and play the narration voice data stream in the structured narration data with the emission source anchor point as the origin.

[0018] Secondly, embodiments of the present invention provide an immersive interactive device for appreciating artifacts, used to implement an immersive interactive method for appreciating artifacts, the device comprising: The interactive tracking unit is used in the immersive digital experience cabin to collect the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bone nodes, parse the gesture commands, and predict and generate the three-dimensional spatial viewpoint parameter set for the next moment. The 3D UV addressing unit is used to obtain UV texture coordinates by performing ray intersection in the virtual engine scene according to the 3D spatial viewpoint parameter set, delineate the 3D region of interest, and retrieve and generate a candidate tile set and a tile topology feature matrix. The optimization scheduling unit is used to solve the global optimal tile loading sequence based on the candidate tile set and the tile topology feature matrix, using an improved tornado optimization algorithm based on a pre-constructed fitness function. The fusion rendering unit is used to perform naked-eye light field rendering and three-dimensional spatial multimodal superposition on the globally optimal tile loading sequence, and present the fusion picture of objects and suspended interpretation on the naked-eye 3D cabin screen of the immersive digital experience cabin.

[0019] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of an immersive interactive method for appreciating objects, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional units of an immersive interactive device for appreciating objects provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] The present invention will be further described below with reference to the accompanying drawings.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0025] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0026] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for an immersive interactive device for appreciating objects.

[0028] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the immersive interactive device for object appreciation stored in the memory 1005 through the processor 1001 and executes the immersive interactive method for object appreciation provided in the embodiment of the present invention.

[0029] Reference Figure 2 The present invention provides an immersive interactive method for appreciating objects, the method comprising: S201: In the immersive digital experience cabin, the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bones are collected, the gesture commands are analyzed, and the three-dimensional spatial viewpoint parameter set for the next moment is predicted and generated. S202: Based on the three-dimensional spatial viewpoint parameter set, perform ray intersection to obtain UV texture coordinates in the virtual engine scene, delineate the three-dimensional region of interest, and retrieve and generate a candidate tile set and a tile topology feature matrix. S203: Based on the candidate tile set and the tile topology feature matrix, and using the improved tornado optimization algorithm based on the pre-constructed fitness function, the globally optimal tile loading sequence is output. S204: Perform naked-eye light field rendering and three-dimensional spatial multimodal superposition on the globally optimal tile loading sequence, and present the fusion image of the object and the suspended interpretation on the naked-eye 3D cabin screen of the immersive digital experience cabin.

[0030] The technical solution provided in this application has at least the following beneficial effects: A Kalman filter based on a uniform motion model combined with real-time perturbations is introduced to predict the dual pupil center, gaze direction, and pinch scaling gain in a unified priori manner. This allows for accurate deduction of the viewpoint parameter set for the next frame from the current frame, enabling subsequent ray intersection and texture loading to be performed one time step ahead. This fundamentally eliminates the visual dragging and dizziness in naked-eye 3D environments, as well as the visual lag caused by head movements and gestures. For high-precision object textures, a fitness function is constructed with three major constraints: visual area coverage, network bandwidth load, and memory fluctuation. This is achieved by introducing Logistic chaotic mapping initialization and incorporating Particle Swarm optimization. The improved Tornado Optimization (PSO) mechanism calculates the globally optimal tile loading sequence that is absolutely safe and has the highest image quality in an extremely short time, eliminating frame drops and memory overflow crashes. In the light field rendering stage, it not only achieves a true naked-eye 3D effect based on micro-viewpoint sub-image synthesis, but also directly queries the object's 3D knowledge graph through the coordinates of the ray hit point, instantiates the structured explanation data into a UI floating panel, and accurately overlays it on the 3D hit point position, realizing a deep fusion appreciation experience of "what you see is what you know".

[0031] In one alternative implementation, within an immersive digital experience cabin, the user's binocular 3D coordinates and hand skeletal node coordinates are collected. Gesture commands are parsed, and a set of 3D spatial viewpoint parameters for the next moment is predicted and generated, including: S2011: In the immersive digital experience cabin, a physical world coordinate system is established with the center of the bottom surface of the naked-eye 3D cabin as the origin, the direction in which the user faces the screen as the positive Z-axis, and the vertical upward direction of gravity as the positive Y-axis. S2012: Uses an eye-tracking module to collect the three-dimensional coordinates of the user's pupils, and uses a motion-sensing module to collect the coordinates of the user's hand skeletal nodes; In this embodiment, the eye-tracking module captures the reflected infrared light spot of the user's pupil in real time, and calculates the three-dimensional coordinates of the left pupil and the right pupil through a stereo vision algorithm. The hand bone node coordinates include the three-dimensional coordinates of 21 bone nodes of the user's hand. S2013: Extract the thumb and index fingertip coordinates from the hand skeleton node coordinates, calculate their Euclidean distance and pinch rate, perform gesture semantic parsing, and if the air focus zoom command is triggered, calculate the virtual camera's zoom gain coefficient using a linear or nonlinear mapping function based on the absolute value of the pinch rate, and proceed to the next step. The formula is: In the formula, For a moment l The Euclidean distance; The coordinates of the thumb tip and the index finger tip are given in the hand bone node coordinates. For a moment The Euclidean distance; For a moment l kneading rate; The unit time step; l For time indication; S2014: If both the Euclidean distance and the kneading rate are greater than the corresponding thresholds, i.e. This triggers the air focus zoom command, and adjusts the zoom based on the pinch rate. The absolute value mapping is used to calculate the scaling gain coefficient of the virtual camera, where, The threshold values ​​are the Euclidean distance threshold and the kneading rate threshold. S2015: Based on the three-dimensional coordinates of both pupils, calculate the coordinates of the center point of both pupils, and construct the gaze direction vector based on the coordinates of the center point of both pupils, the coordinates of the thumb tip, and the coordinates of the index finger tip. The formula is as follows: In the formula, For a moment l The coordinates of the center point of the two pupils; For a moment l The three-dimensional coordinates of the left and right pupils in the dual pupil three-dimensional coordinate system; For a moment l The coordinates of the spatial center point of the kneading action; For a moment l The direction vector of the line of sight; It provides functions for three-dimensional vector operations, ensuring that the user's gaze and focus are perfectly aligned. S2016: Perform Kalman filtering dynamic prediction on the three-dimensional spatial viewpoint parameter vector composed of the coordinates of the center point of the two pupils, the gaze direction vector, and the scaling gain coefficient to obtain the three-dimensional spatial viewpoint parameter set for the next moment. The three-dimensional spatial viewpoint parameter set includes the predicted coordinates of the center point of the two pupils, the predicted gaze direction vector, and the predicted scaling gain coefficient.

[0032] In one optional implementation, a Kalman filter is used to dynamically predict the three-dimensional spatial viewpoint parameter vector, which consists of the coordinates of the dual pupil center points, the gaze direction vector, and the scaling gain coefficient, to obtain the three-dimensional spatial viewpoint parameter set for the next time step, including: S20161: Integrate the current coordinates of the dual pupil center points, the gaze direction vector, and the scaling gain coefficient to obtain the three-dimensional spatial viewpoint parameter vector. ,in, T It is the transpose symbol. For a moment l The scaling gain factor; S20162: Within a unit time step, assuming that the user's head movement, gaze rotation, and gesture zoom are all uniform motions, construct the state transition matrix based on the uniform motion model, as follows: In the formula, This is the state transition matrix; It is a 3-order identity matrix; The unit time step; the top left corner This indicates that the eye position in the next moment is determined by the current position; the upper right corner This indicates that within a unit time step, the eye position will shift at a speed relative to the current line of sight, with the middle... This indicates that the line of sight will remain unchanged in the next moment (assuming uniform speed), and the 1 in the lower right corner indicates that the scaling gain will remain unchanged in the next moment. S20163: Based on the uniform motion model, the control input vector and control matrix are defined as follows: In the formula, For a moment l The control input vector is the acceleration or rate of change actually measured by the sensor, i.e., the real-time disturbance that breaks the uniform velocity assumption. For control matrix; For a moment l The three-dimensional acceleration of head movement (obtained by position difference between two consecutive frames), the angular velocity vector of gaze rotation (obtained by cross product of gaze vectors between two consecutive frames), and the scaling gain change caused by hand pinching. T It is the transpose symbol; S20164: Based on the state transition matrix, control input vector, and control matrix, execute the first prediction equation to calculate the prior state estimate and corresponding prior error covariance matrix of the 3D viewpoint parameter vector at the next time step. The formula is: In the formula, The three-dimensional viewpoint parameter vector at time... l +1 prior state estimate; The three-dimensional viewpoint parameter vector at time... l The posterior state; For a moment l +1 to the prior error covariance matrix; The process noise covariance matrix; T It is the transpose symbol; For a moment l The posterior covariance; S20165: At the next moment, collect the raw measurement values ​​from the eye-tracking module and the motion-sensing module, and package them into the observation vector for the next moment. ,in, For a moment l+ The coordinates of the center point of the two pupils, 1 For a moment l+ The line-of-sight vector of 1, For a moment l+ Scaling gain factor of 1; S20166: Construct the observation matrix and measurement noise covariance matrix, and calculate the Kalman gain using the following formula: In the formula, For a moment l+ Kalman gain of 1; The observation matrix; To measure the noise covariance matrix; T It is the transpose symbol; S20167: Based on the Kalman gain, the prior state estimate and the observation vector are weighted and fused to obtain the optimal posterior estimate for the next time step, as shown in the formula: In the formula, For a moment l+ The optimal posterior estimate of 1; S20168: Update the error confidence matrix to prepare for the next iteration. The formula is: In the formula, For a moment l+ The optimal posterior estimate of 1; For a moment l+ The posterior covariance of 1; It is the identity matrix; S20169: From the optimal posterior estimate The three-dimensional spatial viewpoint parameter set for the next moment is extracted from the package. ,in, For a moment l+ 1. Predicted coordinates of the center points of both pupils. For a moment l+ The predicted gaze direction vector of 1. For a moment l+ The prediction scaling gain coefficient is 1.

[0033] In one optional implementation, based on the three-dimensional spatial viewpoint parameter set, ray intersection is performed in the virtual engine scene to obtain UV texture coordinates, a three-dimensional region of interest is defined, and a candidate tile set and a tile topological feature matrix are retrieved and generated, including: S2021: In the virtual engine scene, a three-dimensional ray is emitted along the direction of the predicted line of sight vector, with the predicted dual pupil center point coordinates in the three-dimensional space viewpoint parameter set as the ray origin. S2022: Using a GPU-accelerated hierarchical bounding box traversal algorithm, calculate the nearest intersection point between the 3D ray and the 3D mesh of the object in the naked-eye 3D cabin, and obtain the 3D coordinates of the nearest intersection point in the local coordinate system of the model and the index of the hit triangle. In this embodiment, in the GPU computing shader of the virtual engine (such as UE5), a three-dimensional ray is emitted along the predicted line of sight vector with the predicted center point of the two pupils as the ray starting point. The hierarchical bounding box acceleration structure built at the bottom layer of the engine is used to skip irrelevant meshes and quickly locate the nearest triangular face and its intersection point of the three-dimensional mesh of the object with logarithmic time complexity. S2023: Read the three vertex data of the triangle corresponding to the index of the hit triangle and their bound texture coordinates. Use the 3D space volume ratio to calculate the weight of the 3D coordinates of the intersection point relative to the centroid coordinates of the triangle. Then, use the interpolation formula to back-calculate the UV texture coordinates of the hit point in the 2D texture space. The formula is: In the formula, These are the coordinate values ​​in a two-dimensional texture space, forming the UV texture coordinates. ; The weights of the three-dimensional coordinates of the intersection point relative to the centroid coordinates of the triangular face; These are texture coordinate values, representing the texture coordinates of the three vertices of the triangle face. ; S2024: Based on the predicted scaling gain coefficients in the three-dimensional spatial viewpoint parameter set, calculate the physical size of a single screen pixel in UV space at the naked-eye 3D cabin screen resolution, and calculate the side length of the rectangular region corresponding to the three-dimensional region of interest, using the following formula: In the formula, The physical size of a single screen pixel in UV space; To predict the scaling gain factor; This is a preset texture scaling base constant; The total number of pixels on the naked-eye 3D cabin screen in a single direction (e.g., horizontal resolution 1920). The rectangular region corresponding to the 3D region of interest is the side length. Based on the predicted scaling gain coefficient, the size of "one screen physical pixel" in the UV 2D texture space at the current viewing distance is dynamically calculated. The larger the user zooms in, the smaller the UV size corresponding to a single pixel, and the higher the required sharpness. S2025: Using UV texture coordinates Using the geometric center as the boundary, the rectangular region corresponding to the 3D region of interest is extended along the positive and negative directions of the horizontal and vertical axes of the UV plane to define a 3D region of interest with a fixed aspect ratio. S2026: Calculate the target texture layer level based on the predicted scaling gain coefficients in the aforementioned 3D spatial viewpoint parameter set, using the following formula: In the formula, The target texture layer level is the theoretically best matching texture clarity level that will not waste video memory under the current prediction scaling. To predict the scaling gain factor; This is the base level index of the pyramid, representing the lowest clearing layer level used by the server when the 3D object model is viewed from the default initial perspective (not zoomed in) (usually set to 0). The function is a floor function. Since texture levels can only be discrete integers (it is impossible for a texture to be layer 1.31.3), floor function must be used to ensure that a definite index is obtained. A "tile" is a preprocessed, regularly segmented, on-demand loaded image or data block. Since the texture adopts a pyramid structure (e.g., level 0 is the lowest resolution, and the resolution doubles at each level), the target texture level is floor functioned to calculate the target texture level that neither wastes video memory nor fails to meet the current scaling resolution requirements. S2027: Traverse the tiles at the target layer level and adjacent layers in the tile server, extract several tiles that spatially overlap with the 3D region of interest, and form a candidate tile set; S2028: Extract the center UV coordinates, tile data volume, and level in the pyramid for each tile in the candidate tile set, and construct the tile topology feature matrix.

[0034] In one optional implementation, based on the candidate tile set and the tile topology feature matrix, and using a pre-constructed fitness function, an improved tornado optimization algorithm is employed to solve for the globally optimal tile loading sequence, including: S2031: Encode the tile loading priority vector of all candidate tiles in the candidate tile set into the position vector of an individual in the improved tornado optimization algorithm, and construct the corresponding fitness function by combining it with the tile topology feature matrix; S2032: Use Logistic mapping to generate chaotic sequences, and map the chaotic sequences to the integer range of individuals in the improved tornado optimization algorithm through carrier mapping and remove duplicates to generate several random integer permutations as the initial individuals, forming the initial population; The formula is: In the formula, For the first n+ 1. n There are several chaotic variables whose values ​​range from [0, 1]. The stability coefficient is typically 4. This sequence is ergodic and random, ensuring that the initial population is uniformly distributed in the solution space, avoiding getting trapped in local optima, which is superior to traditional random initialization. n Indicator of chaotic variables; In the formula, For the first i A continuous real-valued vector; For the first i A vector of chaotic variables, comprising the same number of chaotic variables as the number of candidate tiles in the candidate tile set; These are the upper and lower bounds of the integer interval; i For individual indicators; t This represents the current iteration number; For continuous real value vectors Perform a round-down operation to obtain the rounded result. And a first-hit retention strategy is used to round the result. After deduplication and removing duplicate integer values, the remaining integers will be... The arrangement is the first in the initial population. i An initial individual; if the number of integers in the deduplicated and rounded result does not meet the requirement for the number of candidate tiles, then the next set of chaotic variables is used to continue mapping and rounding until the requirement for the number of candidate tiles is met; In the formula, For the initial population, the first i The initial velocity of each initial individual; A random number in the interval [0,1]. This represents the maximum speed. S2033: Using the fitness function, calculate the fitness value of each initial individual in the initial population, and based on the fitness value, determine the optimal position of each initial individual and the global optimal individual in the initial population. S2034: Introduces a convergence factor and PSO mechanism, and updates the position of the initial population or the population updated in the previous iteration based on the individual optimal position and the global optimal individual to obtain the updated population. S2035: Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; S2036: Repeatedly update the position of the population. When the current iteration reaches the maximum iteration count or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual at the current iteration count. S2037: Decode the position vector of the globally optimal individual to obtain the globally optimal tile loading sequence.

[0035] In one alternative implementation, the fitness function is formulated as follows: In the formula, For the first i The fitness value of an individual; For the first i The total number of tiles in the candidate tile loading priority vector corresponding to an individual; i For individual indicators; j For tile indication; The tile topological feature matrix is ​​the first... i The first individual j The tile data volume, which is directly derived from the tile data volume in the 3rd column of the tile topology feature matrix; The estimated network bandwidth between the eye-tracking module and the motion-sensing module and the local edge rendering server within the immersive digital experience cabin; This is the single-frame rendering time interval of the rendering engine (e.g., approximately 0.0167 seconds at 60fps). This represents the current available remaining video memory capacity of the GPU; The region of interest is a three-dimensional region. This is the area calculation function; For the first i The first individual j The two-dimensional geometric rectangular area occupied by the tile on the UV plane; This is the union operation term for the two-dimensional geometric rectangular region occupied by the tile on the UV plane; The symbol for the intersection of sets; For fitness weighting coefficients; in naked-eye 3D interaction, visual impairment is absolutely unacceptable, therefore It is usually given the highest weight (e.g.) =1000), as a "veto item," memory overflow will cause the program to crash directly, which is a bottom-line error. Assign the second largest weight (e.g.) =500), a slight loading delay is tolerable for users (it manifests as a slightly blurry image that then becomes clearer), therefore Assign minimum weight (e.g.) =10); The system includes network bandwidth load penalty (an extremely severe penalty is imposed if the total data volume of the first few tiles in the sequence divided by the single frame time exceeds the estimated network bandwidth limit), memory fluctuation penalty (a second-largest penalty is imposed if the total data volume exceeds the currently available remaining GPU memory), and actual area coverage gain (the area ratio of the intersection of the union of tiles in the loading sequence and the region of interest (ROI) is calculated).

[0036] In one optional implementation, a convergence factor and a PSO mechanism are introduced, and the initial population or the population updated in the previous iteration is updated based on the individual optimal position and the globally optimal individual, resulting in an updated population, including: S20341: Divide the initial population or the population updated in the previous iteration into the Tornado population with the highest fitness value (top 20%), the Thunderstorm population with the middle fitness value (top 80%-21%), and the Storm population with the lowest fitness value (the remaining individuals). S20342: Introducing a convergence factor and a PSO mechanism, based on the individual optimal position and the globally optimal individual, the tornado population undergoes a fine-grained search around the globally optimal individual to obtain an updated first population. The formula is as follows: In the formula, For the first t+ The first group of species updated in one iteration i The update speed of the first body in the update process; For the first t In the next iteration of the tornado population, the first i The speed of each individual tornado during the initial iteration. The initial velocity of the individual tornado; For the first t The iteration of the ... i The historical best position of an individual tornado; Accelerate one's own cognition; The acceleration coefficient of social cognition; A random number in the interval [0,1]. For the first tThe globally optimal individual in the next iteration; For the first t The convergence factor of the next iteration; For the first t+ The first group of species updated in one iteration i The first updated entity; For the first t The iteration of the ... i Individual tornadoes; t This is an indicator of the number of iterations. In the formula, These are the maximum and minimum values ​​of the convergence factor; This represents the maximum number of iterations. S20343: Introducing a convergence factor, PSO mechanism, and tornado rotation term, based on the individual optimal position and the globally optimal individual, the thunderstorm population explores and develops around the globally optimal individual, resulting in an updated second population. The formula is: In the formula, For the first t+ The second group updated in 1st iteration i The update speed of the second body; For the first t In the next iteration of the thunderstorm population, the first i The speed of each individual thunderstorm, in the first iteration The initial velocity of the individual thunderstorm; For the first t The iteration of the ... i The historical best position of an individual thunderstorm; Accelerate one's own cognition; The acceleration coefficient of social cognition; A random number in the interval [0,1]. For the first t The globally optimal individual in the next iteration; For the first t The convergence factor of the next iteration; For the first t+ The second group updated in 1st iteration i A newer second individual; For the first t The iteration of the ... i Individual thunderstorms; e The base of the exponent; For tornado rotation; The rotation factor; The rotation intensity increases with the number of iterations; In the formula, It is the rotational reinforcement constant; S20344: Introducing a convergence factor, PSO mechanism, and stochastic traction term, based on the individual optimal position and the globally optimal individual, the storm population conducts a global exploration around the globally optimal individual, resulting in an updated third population. The formula is as follows: In the formula, For the first t+ The third group in the first iteration of the update i The update speed of the third body; For the first t In the next iteration of the storm population, the first i The speed of each individual storm, during the initial iteration. The initial velocity of the individual storm; For the first t The iteration of the ... i The historical best position of an individual storm; Accelerate one's own cognition; The acceleration coefficient of social cognition; A random number in the interval [0,1]. For the first t The globally optimal individual in the next iteration; For the first t The convergence factor of the next iteration; For the first t+ The third group in the first iteration of the update i A third updated entity; For the first t The iteration of the ... i Individual storms; Let be a random perturbation vector in the interval [0,1]. S20345: Integrate the updated first population, the updated second population, and the updated third population to obtain the updated population.

[0037] In one optional implementation, the globally optimal tile loading sequence is rendered using naked-eye light field rendering and superimposed with three-dimensional spatial multimodal rendering, and the fused image of the object and its suspended interpretation is presented on the naked-eye 3D screen of the immersive digital experience cabin, including: S2041: Based on the optimal fitness value corresponding to the global optimal tile loading sequence, if the optimal fitness value is greater than the preset safety threshold, it is determined that there is a risk of frame drop, the prediction scaling gain coefficient in the three-dimensional spatial viewpoint parameter set is reduced, and the three-dimensional region of interest delineation step is returned. In this embodiment, when there is a risk of frame drops, a "circuit breaker mechanism" is triggered, which actively reduces the prediction scaling gain coefficient by one level (for example, from 5.0 back to 3.0), and forces a jump back to the three-dimensional region of interest delineation step to redefine a smaller and easier-to-load ROI in order to maintain the smoothness of the screen. S2042: Analyze the globally optimal tile loading sequence and initiate network fetch requests to the tile server in order of optimal tile loading priority; S2043: Based on the network fetch request, use the GPU memory controller to read the tile data volume of each tile in the globally optimal tile loading sequence, perform pre-allocation and page locking of continuous virtual memory in the memory pool, and complete the tile loading; S2044: During the GPU pixel shading stage, read the 3D region of interest and extract the pixel UV texture coordinates of each currently rendered pixel on the object's 3D mesh; S2045: Calculate the tile row and column index that the pixel should currently hit based on the pixel UV texture coordinates and the target texture level corresponding to the predicted scaling gain coefficient. S2046: If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has been loaded, then the pixel is determined to belong to the high-optimal loading region, and full-resolution high dynamic range physically-based rendering (PBR) material sampling is performed on the pixel. S2047: If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has not been loaded, then it is determined that the high-definition tile corresponding to the pixel is not ready, and the low-level blur map resident in the video memory is directly sampled for the pixel. S2048: If the tile row and column index does not exist in the global optimal tile loading sequence, then the pixel is determined not to belong to the core focus area, and the low-level blur map resident in the video memory is directly sampled for the pixel. S2049: Traverse all pixels of the object's 3D mesh, repeat the GPU pixel shading process, and combine the predicted dual pupil center point coordinates in the 3D spatial viewpoint parameter set to generate several micro-viewpoint sub-images around the predicted dual pupil center point coordinates using a light field synthesizer. S20410: Synthesize several micro-view sub-images to obtain a micro-view composite image of the object, and output the micro-view composite image to the naked-eye 3D cabin screen of the immersive digital experience cabin to complete naked-eye light field rendering. S20411: Extract the three-dimensional coordinates of the nearest intersection point, and use the three-dimensional coordinates of the intersection point as the query key to input into the local pre-built three-dimensional knowledge graph database of objects; S20412: If there is a matching entity for the query key, then retrieve the structured explanation data corresponding to that matching entity. S20413: In the virtual engine scene, instantiate the corresponding UI floating panel, render the structured explanation data onto the UI floating panel to obtain the floating interpretation panel, and overlay the floating interpretation panel onto the micro-view synthesized image of the naked-eye 3D cabin screen to complete the three-dimensional space multimodal overlay.

[0038] In one alternative implementation, the structured explanation data includes the explanatory text string, highlighted material attribute parameters, and the explanation voice data stream; A-1: The explanatory text string in the structured explanatory data is drawn onto the preset layout area of ​​the UI floating panel using a text rendering component; A-2: Extract the self-illuminating color and transparency coefficient of the highlight material attribute parameters in the structured explanation data, and apply the self-illuminating color and transparency coefficient to the background mask and / or the underlying three-dimensional surface of the UI floating panel to generate a floating interpretation panel with visual focus guidance effect. A-3: Register the three-dimensional spatial coordinates of the suspended interpretation panel as a spatial audio emission source anchor point, call the spatial audio module, and play the narration voice data stream in the structured narration data with the emission source anchor point as the origin, so that when the user moves in the immersive digital experience cabin, he / she can perceive the spatial sound field change that matches the position of the suspended interpretation panel.

[0039] This invention also provides an immersive interactive device 300 for appreciating objects, see reference. Figure 3 The device may include the following units: The interactive tracking unit is used in the immersive digital experience cabin to collect the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bone nodes, parse the gesture commands, and predict and generate the three-dimensional spatial viewpoint parameter set for the next moment. The 3D UV addressing unit is used to obtain UV texture coordinates by performing ray intersection in the virtual engine scene according to the 3D spatial viewpoint parameter set, delineate the 3D region of interest, and retrieve and generate a candidate tile set and a tile topology feature matrix. The optimization scheduling unit is used to solve the global optimal tile loading sequence based on the candidate tile set and the tile topology feature matrix, using an improved tornado optimization algorithm based on a pre-constructed fitness function. The fusion rendering unit is used to perform naked-eye light field rendering and three-dimensional spatial multimodal superposition on the globally optimal tile loading sequence, and present the fusion picture of objects and suspended interpretation on the naked-eye 3D cabin screen of the immersive digital experience cabin.

[0040] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the immersive interactive method for appreciating objects according to the present invention.

[0041] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0042] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0043] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the immersive interactive method for appreciating objects according to embodiments of the present invention.

[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0049] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An immersive interactive method for appreciating artifacts, characterized in that, The method includes: In the immersive digital experience cabin, the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bones are collected, the gesture commands are analyzed, and the three-dimensional spatial viewpoint parameter set for the next moment is predicted and generated. Based on the three-dimensional spatial viewpoint parameter set, ray intersection is performed in the virtual engine scene to obtain UV texture coordinates, three-dimensional regions of interest are delineated, and candidate tile sets and tile topological feature matrices are retrieved and generated. Based on the candidate tile set and tile topology feature matrix, and using a pre-constructed fitness function, an improved tornado optimization algorithm is employed to solve the problem and output the globally optimal tile loading sequence. The globally optimal tile loading sequence is rendered using naked-eye light field rendering and superimposed with three-dimensional spatial multimodal rendering, and the fused image of objects and suspended interpretation is presented on the naked-eye 3D cabin screen of the immersive digital experience cabin.

2. The immersive interactive method for appreciating artifacts according to claim 1, characterized in that, In the immersive digital experience cabin, the 3D coordinates of the user's dual pupils and the coordinates of the hand skeleton nodes are collected. Gesture commands are analyzed, and a set of 3D spatial viewpoint parameters for the next moment is predicted and generated, including: In the immersive digital experience cabin, a physical world coordinate system is established with the center of the bottom surface of the naked-eye 3D cabin as the origin; The eye-tracking module is used to collect the three-dimensional coordinates of the user's pupils, and the motion sensing module is used to collect the coordinates of the user's hand skeletal nodes. Extract the coordinates of the thumb tip and index finger tip from the hand bone node coordinates, calculate the Euclidean distance and pinching rate between them, perform gesture semantic parsing, and if the air focus zoom command is triggered, calculate the zoom gain coefficient of the virtual camera and proceed to the next step. If both the Euclidean distance and the pinch rate are greater than the corresponding threshold, the remote focus scaling command is triggered, and the scaling gain coefficient of the virtual camera is calculated based on the absolute value of the pinch rate. Based on the three-dimensional coordinates of the two pupils, calculate the coordinates of the center point of the two pupils, and construct the gaze direction vector based on the coordinates of the center point of the two pupils, the coordinates of the thumb tip, and the coordinates of the index finger tip; Kalman filtering is used to dynamically predict the three-dimensional spatial viewpoint parameter vector, which consists of the coordinates of the center points of the two pupils, the gaze direction vector, and the scaling gain coefficient, to obtain the three-dimensional spatial viewpoint parameter set for the next moment. The three-dimensional spatial viewpoint parameter set includes the predicted coordinates of the center points of the two pupils, the predicted gaze direction vector, and the predicted scaling gain coefficient.

3. The immersive interactive method for appreciating artifacts according to claim 2, characterized in that, Kalman filtering is used to dynamically predict the 3D spatial viewpoint parameter vector, which consists of the coordinates of the center points of the two pupils, the gaze direction vector, and the scaling gain coefficient, to obtain the 3D spatial viewpoint parameter set for the next time step, including: By integrating the current coordinates of the center points of the two pupils, the gaze direction vector, and the scaling gain coefficient, a three-dimensional spatial viewpoint parameter vector is obtained. Within a unit time step, assuming that the user's head movement, gaze rotation, and gesture zoom are all uniform motions, construct a state transition matrix based on the uniform motion model; Based on the uniform motion model, the control input vector and control matrix are defined. Based on the state transition matrix, control input vector, and control matrix, execute the first prediction equation to calculate the prior state estimate of the three-dimensional spatial viewpoint parameter vector at the next time step and the corresponding prior error covariance matrix. When the next moment arrives, the raw measurement values ​​of the eye-tracking module and the motion sensing module are collected and packaged into the observation vector for the next moment; Construct the observation matrix and measurement noise covariance matrix, and calculate the Kalman gain; Based on the Kalman gain, the prior state estimate and the observation vector are weighted and fused to obtain the optimal posterior estimate for the next time step. Update the error confidence matrix to prepare for the next iteration; The three-dimensional spatial viewpoint parameter set for the next time step is extracted from the optimal posterior estimate.

4. The immersive interactive method for appreciating artifacts according to claim 3, characterized in that, Based on the aforementioned 3D spatial viewpoint parameter set, in the virtual engine scene, ray intersection is performed to obtain UV texture coordinates, a 3D region of interest is defined, and a candidate tile set and tile topological feature matrix are retrieved and generated, including: In the virtual engine scene, a three-dimensional ray is emitted along the direction of the predicted line of sight vector, with the coordinates of the predicted dual pupil center point in the three-dimensional spatial viewpoint parameter set as the ray origin. Using a GPU-accelerated hierarchical bounding box traversal algorithm, the nearest intersection point of the 3D ray and the 3D mesh of the object in the naked-eye 3D cabin is calculated, and the 3D coordinates of the nearest intersection point and the index of the hit triangle face in the local coordinate system of the model are obtained. Read the data of the three vertices of the triangle corresponding to the index of the hit triangle and their bound texture coordinates. Use the 3D space volume ratio to calculate the weight of the 3D coordinates of the intersection point relative to the centroid coordinates of the triangle. Then, use the interpolation formula to back-calculate the UV texture coordinates of the hit point in the 2D texture space. Based on the predicted scaling gain coefficients in the three-dimensional spatial viewpoint parameter set, the physical size of a single screen pixel in UV space is calculated at the naked-eye 3D cabin screen resolution, and the side length of the rectangular region corresponding to the three-dimensional region of interest is calculated. Using the UV texture coordinates as the geometric center, the rectangular region corresponding to the 3D region of interest is extended in the positive and negative directions of the horizontal and vertical axes of the UV plane to define a 3D region of interest with a fixed aspect ratio. Calculate the target texture layer level based on the predicted scaling gain coefficients in the three-dimensional spatial viewpoint parameter set; Traverse the tiles in the target layer and adjacent layers of the tile server, extract several tiles that spatially overlap with the 3D region of interest, and form a candidate tile set; Extract the center UV coordinates, tile data volume, and level in the pyramid for each tile in the candidate tile set, and construct the tile topology feature matrix.

5. The immersive interactive method for appreciating artifacts according to claim 4, characterized in that, Based on the candidate tile set and tile topological feature matrix, and using a pre-constructed fitness function, an improved tornado optimization algorithm is employed to solve the problem and output the globally optimal tile loading sequence, including: The tile loading priority vector of all candidate tiles in the candidate tile set is encoded into the position vector of an individual in the improved tornado optimization algorithm, and combined with the tile topology feature matrix, the corresponding fitness function is constructed. The chaotic sequence is generated using Logistic mapping, and then mapped to the integer range of individuals in the improved tornado optimization algorithm through carrier mapping and deduplication, generating several random integer permutations as the initial individuals to form the initial population. Using the fitness function, the fitness value of each initial individual in the initial population is calculated, and based on the fitness value, the optimal position of each initial individual and the global optimal individual in the initial population are determined. A convergence factor and a PSO mechanism are introduced, and the positions of the initial population or the population updated in the previous iteration are updated based on the individual optimal position and the global optimal individual to obtain the updated population. Using the fitness function, calculate the fitness value of each updated individual in the updated population, and update the global best individual based on the fitness value; Repeatedly update the position of the population. When the current iteration reaches the maximum iteration count or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the globally optimal individual at the current iteration count. The position vector of the globally optimal individual is decoded to obtain the globally optimal tile loading sequence.

6. The immersive interactive method for appreciating artifacts according to claim 5, characterized in that, The formula for the fitness function is: In the formula, For the first i The fitness value of an individual; For the first i The total number of tiles in the candidate tile loading priority vector corresponding to an individual; i For individual indicators; j For tile indication; The tile topological feature matrix is ​​the first... i The first individual j The tile data volume, which is directly derived from the tile data volume in the 3rd column of the tile topology feature matrix; The estimated network bandwidth between the eye-tracking module and the motion-sensing module and the local edge rendering server within the immersive digital experience cabin; This refers to the single-frame rendering time interval of the rendering engine; This represents the current available remaining video memory capacity of the GPU; The region of interest is a three-dimensional region. This is the area calculation function; For the first i The first individual j The two-dimensional geometric rectangular area occupied by the tile on the UV plane; This is the union operation term for the two-dimensional geometric rectangular region occupied by the tile on the UV plane; The symbol for the intersection of sets; This refers to the fitness weighting coefficient.

7. The immersive interactive method for appreciating artifacts according to claim 6, characterized in that, A convergence factor and a PSO mechanism are introduced, and the initial population or the population updated in the previous iteration is updated based on the individual optimal position and the globally optimal individual, resulting in an updated population, including: The initial population or the population updated in the previous iteration is divided into the tornado population with the highest fitness value, the thunderstorm population with the middle fitness value, and the storm population with the lowest fitness value. By introducing a convergence factor and a PSO mechanism, based on the individual optimal position and the global optimal individual, the tornado population is finely searched around the global optimal individual to obtain an updated first population. By introducing a convergence factor, a PSO mechanism, and a tornado rotation term, and based on the individual optimal position and the global optimal individual, the thunderstorm population explores and develops around the global optimal individual, resulting in a newer second population. By introducing a convergence factor, a PSO mechanism, and a stochastic traction term, the storm population is made to explore the world around the globally optimal individual based on the individual's optimal position and the globally optimal individual, thus obtaining an updated third population. By integrating the updated first population, the updated second population, and the updated third population, an updated population is obtained.

8. The immersive interactive method for appreciating artifacts according to claim 7, characterized in that, The globally optimal tile loading sequence is rendered using naked-eye light field rendering and superimposed with three-dimensional spatial multimodal rendering. The resulting image, a fusion of the object and its suspended interpretation, is then displayed on the naked-eye 3D screen of the immersive digital experience cabin, including: Based on the optimal fitness value corresponding to the globally optimal tile loading sequence, if the optimal fitness value is greater than the preset safety threshold, it is determined that there is a risk of frame drop, the prediction scaling gain coefficient in the three-dimensional spatial viewpoint parameter set is reduced, and the three-dimensional region of interest delineation step is returned. The globally optimal tile loading sequence is parsed, and network fetch requests are sent to the tile server in order of optimal tile loading priority. Based on the network fetch request, the GPU memory controller reads the tile data volume of each tile in the globally optimal tile loading sequence, performs pre-allocation and page locking of continuous virtual memory in the memory pool, and completes the tile loading. During the GPU pixel shading stage, the 3D region of interest is read, and the pixel UV texture coordinates of each currently rendered pixel on the object's 3D mesh are extracted. Calculate the tile row and column index that the pixel should currently hit based on the pixel UV texture coordinates and the target texture level corresponding to the predicted scaling gain coefficient; If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has been loaded, then the pixel is determined to belong to the high-optimal loading region, and full-resolution high dynamic range PBR material sampling is performed on the pixel. If the tile row and column index exists in the global optimal tile loading sequence, and the corresponding tile in the global optimal tile loading sequence has not been fully loaded, then it is determined that the high-definition tile corresponding to the pixel is not yet ready, and the low-level blur map resident in the video memory is directly sampled for the pixel. If the tile row and column index does not exist in the global optimal tile loading sequence, then the pixel is determined not to belong to the core focus area, and the low-level blur map resident in the video memory is directly sampled for the pixel. Traverse all pixels of the object's 3D mesh, repeat the GPU pixel shading process, and combine the predicted dual pupil center point coordinates from the 3D spatial viewpoint parameter set to generate several micro-viewpoint sub-images around the predicted dual pupil center point coordinates using a light field synthesizer. Several micro-perspective sub-images are synthesized to obtain a micro-perspective composite image of the object, and the micro-perspective composite image is output to the naked-eye 3D cabin screen of the immersive digital experience cabin to complete naked-eye light field rendering. Extract the 3D coordinates of the nearest intersection point, and use these 3D coordinates as the query key to input them into a pre-built 3D knowledge graph database of artifacts. If a matching entity exists for the query key, then retrieve the structured explanation data corresponding to that matching entity. In the virtual engine scene, the corresponding UI floating panel is instantiated, and the structured explanation data is rendered onto the UI floating panel to obtain the floating interpretation panel. The floating interpretation panel is then superimposed onto the micro-view synthesized image on the naked-eye 3D cabin screen to complete the three-dimensional space multimodal superposition.

9. The immersive interactive method for appreciating artifacts according to claim 8, characterized in that, The structured explanation data includes the explanatory text string, highlighted material attribute parameters, and the explanation voice data stream; The explanatory text strings in the structured explanatory data are drawn onto the preset layout area of ​​the UI floating panel using a text rendering component; Extract the self-illuminating color and transparency coefficient of the highlighted material attribute parameters from the structured explanation data, and apply the self-illuminating color and transparency coefficient to the background mask and / or the underlying 3D surface of the UI floating panel to generate a floating interpretation panel with visual focus guidance effect. Register the three-dimensional spatial coordinates of the floating interpretation panel as a spatial audio emission source anchor point, call the spatial audio module, and play the narration voice data stream in the structured narration data with the emission source anchor point as the origin.

10. An immersive interactive device for appreciating objects, used to implement the immersive interactive method for appreciating objects as described in any one of claims 1-9, characterized in that, The device includes: The interactive tracking unit is used in the immersive digital experience cabin to collect the three-dimensional coordinates of the user's binoculars and the coordinates of the hand bone nodes, parse the gesture commands, and predict and generate the three-dimensional spatial viewpoint parameter set for the next moment. The 3D UV addressing unit is used to obtain UV texture coordinates by performing ray intersection in the virtual engine scene according to the 3D spatial viewpoint parameter set, delineate the 3D region of interest, and retrieve and generate a candidate tile set and a tile topology feature matrix. The optimization scheduling unit is used to solve the global optimal tile loading sequence based on the candidate tile set and the tile topology feature matrix, using an improved tornado optimization algorithm based on a pre-constructed fitness function. The fusion rendering unit is used to perform naked-eye light field rendering and three-dimensional spatial multimodal superposition on the globally optimal tile loading sequence, and present the fusion picture of objects and suspended interpretation on the naked-eye 3D cabin screen of the immersive digital experience cabin.

Images