Real-time feedback method for a drift river game system based on holographic image interaction

Abstract

The application discloses a real-time feedback method of a drifting river game system based on holographic image interaction, relates to the technical field of virtual interaction, and comprises the following steps: according to ambient light data, calling a graphic rendering tool to perform real-time rendering on a virtual interactive object that has collided, and calculating the pixel-level color and brightness value of the virtual interactive object to generate a target holographic image frame; according to collision point coordinate information, calculating the mechanical effect when a virtual interactive ray collides with the virtual interactive object, mapping the mechanical effect into a physical disturbance parameter of a drifting river play vehicle, generating a physical feedback control instruction, sending the physical feedback control instruction to a physical feedback device, driving the physical feedback device to generate a physical environment disturbance that is synchronous with the target holographic image frame, and acting on the drifting river play vehicle; and the application realizes the comprehensive effect of deep virtual-real integration, real and credible interaction, and immersive and coherent experience of the drifting river game through an immersive virtual interaction method.

Description

Technical Field

[0001] This invention relates to the field of virtual interaction technology, and in particular to a real-time feedback method for a rafting river game system based on holographic image interaction. Background Technology

[0002] With the continuous breakthroughs in virtual reality, augmented reality, and holographic display methods, immersive interactive entertainment is becoming increasingly widespread. As a classic water activity that combines leisure and fun, river rafting has gradually integrated advanced digital interaction methods in recent years, evolving towards intelligence and interactivity. Currently, mainstream solutions provide dynamic visual content to tourists through high-brightness projection mapping. At the same time, integrating underwater vibration units, directional jet devices, and motion platforms can trigger tactile and kinesthetic stimulation at specific times, enhancing the realism of users' perception of virtual events. This not only improves the interactive dimension and visual expressiveness of entertainment but also provides solid technical support for immersive river rafting game systems.

[0003] Nevertheless, the real-time feedback methods of existing river drifting game systems still have room for improvement. They use fixed animation sequences to respond to user input and cannot generate holographic images based on the interaction position and ambient lighting, resulting in a disconnect between visual feedback and user operation. At the same time, the lack of a collision mechanics mapping mechanism causes the motion state of objects in the holographic image to be mismatched with the disturbances experienced by the actual vehicle, thus ruining the immersive experience. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a real-time feedback method for a holographic image interaction-based river drifting game system to solve the problem of disconnect between visual feedback and user operation.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a real-time feedback method for a holographic image-interactive river drifting game system, comprising:

[0008] Collect 3D point cloud data of the rafting river, and use surface reconstruction algorithm to model the 3D point cloud data of the rafting river into a virtual scene model of the rafting river. At the same time, collect ambient light data on the rafting river and interaction data between tourists and interactive devices.

[0009] Based on the interaction data between tourists and interactive devices, the system identifies tourists' interactive actions and determines the occurrence of interactive events. When an interactive event is determined to have occurred, a virtual interactive ray is generated starting from the position of the interactive device.

[0010] Collision detection is performed between the virtual interactive ray and the virtual interactive objects in the rafting river virtual scene model. When a collision is detected, the coordinate information of the collision point is recorded.

[0011] Based on ambient light data, the graphics rendering tool is invoked to render the virtual interactive objects that collide in real time, and the pixel-level color and brightness values ​​of the virtual interactive objects are calculated to generate target holographic image frames.

[0012] Calculate the mechanical effects when the virtual interactive ray collides with the virtual interactive object based on the collision point coordinates.

[0013] The mechanical effects are mapped to physical disturbance parameters of the rafting vehicle, and physical feedback control commands are generated.

[0014] The physical feedback control command is sent to the physical feedback device, which drives the physical feedback device to generate physical environmental disturbances synchronized with the target holographic image frame, and then acts on the rafting vehicle.

[0015] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based river drifting game system described in this invention, the step of acquiring 3D point cloud data of the river drifting and using a surface reconstruction algorithm to model the 3D point cloud data of the river drifting into a virtual scene model of the river drifting, specifically involves:

[0016] During the initialization phase of the rafting river game, a 3D laser scanner was used to scan the real rafting river channel at multiple stations to obtain 3D point cloud data of the rafting river, and the 3D point cloud data of the rafting river was then registered.

[0017] The registered 3D point cloud data of the drifting river was reconstructed into a 3D mesh model of the drifting river using the Poisson surface reconstruction algorithm.

[0018] The 3D mesh model of the rafting river is simplified by reducing the number of triangular faces while maintaining geometric features. The boundaries of the virtual interaction area and the feasible area are marked on the simplified 3D mesh model of the rafting river to obtain the virtual scene model of the rafting river.

[0019] As a preferred embodiment of the real-time feedback method of the holographic image interaction-based drifting river game system described in this invention, the ambient light data on the drifting river includes light intensity, main direction vector of the light source, and color temperature.

[0020] The interaction data between the tourist and the interactive device includes the spatial pose data of the interactive device and the interaction trigger signal.

[0021] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based river drifting game system described in this invention, the method involves: identifying the interactive actions of tourists and determining the occurrence of interactive events based on the interaction data between tourists and interactive devices; and when an interactive event is determined to have occurred, generating a virtual interactive ray starting from the position of the interactive device. Specifically:

[0022] Based on the interaction data between tourists and interactive devices, the status of interaction trigger signals is continuously monitored. When the interaction trigger signal is detected to change from an invalid state to an valid state, it is determined that an interactive event has occurred.

[0023] After an interactive event occurs, read the spatial pose data of the interactive device;

[0024] The spatial pose data of the interactive device includes position coordinates and orientation angle;

[0025] Using the position coordinates as the origin of the virtual interaction ray and the direction of the angle as the direction of the virtual interaction ray, a virtual interaction ray is instantiated and generated in the virtual scene model of the drifting river.

[0026] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based river drifting game system described in this invention, the method involves: performing collision detection between the virtual interactive ray and the virtual interactive object in the river drifting virtual scene model; and recording the collision point coordinates when a collision is detected. Specifically:

[0027] Collision detection is performed between the virtual interactive ray and the virtual interactive objects in the virtual scene model of the drifting river.

[0028] During collision detection, the mathematical equation of the virtual interaction ray is calculated and its spatial positional relationship with the triangular facet geometry of the virtual interaction object is determined. When the calculation results show that the virtual interaction ray and the triangular facet geometry of the virtual interaction object intersect, a collision is determined to have been detected.

[0029] After a collision occurs, calculate the coordinates of the intersection point between the virtual interactive ray and the triangular facet geometry of the virtual interactive object, and record the intersection point coordinates as the collision point coordinate information.

[0030] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based river drifting game system described in this invention, wherein:

[0031] The process involves using ambient light data to call a graphics rendering tool to render the colliding virtual interactive objects in real time, calculating the pixel-level color and brightness values ​​of the virtual interactive objects, and generating a target holographic image frame. Specifically:

[0032] Calculate the diffuse reflection light component of each pixel on the surface of the virtual interactive object based on the principal direction vector of the light source;

[0033] Based on the diffuse illumination component, the spectral energy distribution of the pixels is adjusted using color temperature. Then, the ambient light occlusion and specular reflection components are integrated using a graphics rendering tool. Finally, the pixel-level color and brightness values ​​of the virtual interactive object surface are calculated pixel by pixel to obtain the target holographic image frame.

[0034] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based drifting river game system described in this invention, the step of calculating the mechanical effect when a virtual interactive ray collides with a virtual interactive object based on the collision point coordinate information specifically includes:

[0035] Collect the normal vector of the surface of the virtual interactive object at the collision point from the collision point coordinate information, and calculate the impulse generated by the collision using the impulse theorem;

[0036] The linear and angular velocity changes of virtual interactive objects are calculated using the impulse generated by collisions to obtain the mechanical effects.

[0037] As a preferred embodiment of the real-time feedback method of the holographic image interaction-based rafting river game system described in this invention, the step of mapping the mechanical effect to the physical disturbance parameters of the rafting river vehicle refers to calculating the physical disturbance parameters that the rafting river vehicle should be subjected to in the virtual water body based on the linear velocity change and angular velocity change of the virtual interactive object in the mechanical effect.

[0038] As a preferred embodiment of the real-time feedback method of the holographic image interaction-based drifting river game system described in this invention, the generation of physical feedback control instructions refers to encoding physical disturbance parameters into instructions that can be recognized and executed by the physical feedback device.

[0039] As a preferred embodiment of the real-time feedback method for the holographic image interaction-based river drifting game system described in this invention, the step of sending physical feedback control commands to a physical feedback device to drive the physical feedback device to generate physical environmental disturbances synchronized with the target holographic image frame and acting on the river drifting vehicle specifically involves:

[0040] The physical feedback control command is sent to the physical feedback device via a wireless communication link;

[0041] After receiving and parsing the physical feedback control command, the physical feedback device generates a wave sequence in the river. After synchronizing the target holographic image frame with the wave sequence, it generates thrust and swaying force on the rafting vehicle.

[0042] The beneficial effects of this invention are as follows: by collecting ambient light data and combining it with a physically based rendering method, a seamless integration of holographic images and real lighting environment is achieved. At the same time, mechanical effects are calculated, a physical causal logic of interactive behavior is established, and virtual mechanical parameters are mapped into adjustable physical disturbance commands. This enables precise conversion and safe adaptation of force feedback between the virtual and real worlds. Through the spatiotemporal synchronization of physical feedback devices and holographic images, an immersive closed-loop experience with high consistency of visual, auditory, and tactile senses is constructed, thereby achieving a comprehensive entertainment effect of deep integration of virtual and real worlds, realistic and credible interaction, and immersive and coherent experience. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a real-time feedback method for a holographic image-based interactive river drifting game system.

[0045] Figure 2 A flowchart for obtaining a virtual scene model of a drifting river.

[0046] Figure 3 A flowchart for obtaining mechanical effects.

[0047] Figure 4 A flowchart for generating a target holographic image frame. Detailed Implementation

[0048] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0051] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a real-time feedback method for a rafting river game system based on holographic image interaction, comprising the following steps:

[0052] S1. Collect 3D point cloud data of the rafting river, and use the surface reconstruction algorithm to model the 3D point cloud data of the rafting river into a virtual scene model of the rafting river. At the same time, collect ambient light data on the rafting river and interaction data between tourists and interactive devices.

[0053] S1.1 During the initialization phase of the rafting river game, a 3D laser scanner is used to scan the real rafting river channel at multiple stations to obtain 3D point cloud data of the rafting river, and the 3D point cloud data of the rafting river is registered.

[0054] It should be noted that during the initialization phase of the rafting river game, multiple scanning stations were deployed along the actual rafting river channel using a 3D laser scanner. At each scanning station, the 3D laser scanner emitted a laser beam and received the reflected signal to acquire 3D point cloud data of the rafting river. After scanning all stations, an iterative nearest-point algorithm was used to register the 3D point cloud data segments of the rafting river at adjacent stations. Specifically, firstly, the 3D point cloud data segments of the rafting river at two adjacent scanning stations were selected as the source point cloud and the target point cloud, and corresponding point pairs were established in the overlapping area of ​​the source point cloud and the target point cloud. By minimizing the sum of squared average distances between corresponding point pairs, the optimal rigid body transformation matrix was calculated, expressed by the formula:

[0055] ;

[0056] in, This represents the optimal rigid body transformation matrix. Indicates the number of corresponding point pairs. Indicates the first point in the target point cloud One point, Indicates the first point in the source cloud One point, Represents the rotation matrix. Represents the translation vector;

[0057] The rigid body transformation matrix includes a rotation matrix and a translation vector, used to transform the source point cloud into the target point cloud's coordinate system. This involves multiplying the coordinates of each 3D point in the source point cloud by the rotation matrix, and then adding the result to the translation vector to obtain the new coordinates in the target point cloud's coordinate system. After processing with the rotation matrix and translation vector, the source and target point clouds are in the same coordinate system.

[0058] The transformed source point cloud and target point cloud undergo a new round of corresponding point search and transformation matrix calculation. The iterative process continues until the maximum number of iterations. After that, the 3D point cloud data of the drifting river from all scanning stations are unified to the global coordinate system through continuous registration.

[0059] It should also be noted that the maximum number of iterations is usually set based on the complexity of the point cloud data and the actual application requirements, with a common range of 20 to 100. When the number of iterations is less than 20, basic convergence cannot be guaranteed and the registration result is unreliable. When the number of iterations is greater than 100, diminishing returns will occur and the computational cost will be high. This usually means that there is a problem with the data or algorithm itself, rather than an insufficient number of iterations.

[0060] S1.2. The registered 3D point cloud data of the drifting river is reconstructed into a 3D mesh model of the drifting river using the Poisson surface reconstruction algorithm.

[0061] It should be noted that all three-dimensional coordinate points are extracted from the registered 3D point cloud data of the drifting river to form an initial point set. A spatial neighborhood (including k nearest neighbors) is selected for each point in the initial point set. Principal component analysis is used to perform feature analysis on the point cloud patch composed of each point and its k nearest neighbors. The covariance matrix of the point cloud patch is calculated and eigenvalue decomposition is performed. The eigenvector corresponding to the smallest eigenvalue is selected as the normal vector direction of this point. The point set composed of the coordinates of all points in the 3D point cloud data of the drifting river is obtained.

[0062] An octree structure in 3D space is constructed for spatial partitioning, and an indicator function is defined at each node of the octree. The problem of reconstructing the surface of the point set is transformed into solving the Poisson equation problem of the least squares difference between the gradient field of the indicator function and the vector field (the vector field composed of the normal vectors of the 3D point cloud data of the drifting river). The Poisson equation is solved to calculate the indicator function value defined at each node of the octree, where the Poisson equation is expressed as:

[0063]

[0064] in, Let Laplace operator be the indicator function scalar field to be solved. Denotes the divergence operator, This represents the vector field formed by the normal vectors of the 3D point cloud data of the drifting river;

[0065] The indicator function values ​​collectively define a continuous scalar field, and the isosurface of the scalar field represents the surface described by the 3D point cloud data of the drifting river.

[0066] The isosurface threshold is set based on the indicator function value; the moving cube algorithm is used to detect the edges that intersect with the isosurface in each cube voxel in 3D space; based on the comparison between the indicator function value at the eight vertices of the cube voxel and the isosurface threshold, the isosurface topology within the current cube voxel is determined by searching a predefined isosurface configuration table; the coordinates of the intersection points of the isosurface and the edges are calculated by linear interpolation on the boundary of the cube voxel, and the intersection points are connected into triangular patches according to the isosurface topology; the triangular patches generated by all cube voxels are combined into a continuous set of triangular patches, i.e., the 3D mesh model of the drifting river.

[0067] It should also be noted that the isosurface threshold is usually set to 0, which is based on the mathematical principle of Poisson reconstruction. If the isosurface threshold is not 0, the extracted isosurface will no longer be a closed manifold, thus destroying the "watertightness" of the 3D mesh model of the drifting river.

[0068] The predefined isosurface configuration table generates 256 possible vertex state configurations by enumerating all internal and external state combinations of the eight vertices of a cube voxel. Utilizing the cube's rotational and mirror symmetry, the 256 vertex state configurations are topologically equivalent, grouping vertex state configurations with the same topological structure into a basic configuration, resulting in 15 unique basic configurations. For each basic configuration, the intersection distribution pattern of the isosurface and the twelve edges of the cube voxel is analyzed, and the connection scheme of the triangular facets is determined based on the intersection distribution pattern. The 15 basic configurations and their corresponding triangular facet connection schemes are encoded and stored in a lookup table.

[0069] S1.3 Simplify the 3D mesh model of the rafting river, reduce the number of triangular faces while maintaining geometric features, mark the boundaries of the virtual interaction area and the feasible area on the simplified 3D mesh model of the rafting river, and obtain the virtual scene model of the rafting river.

[0070] It should be noted that an edge-folding simplification algorithm is used to process the 3D mesh model of the rafting river. This algorithm iteratively calculates the folding cost of all edges in the 3D mesh model. Specifically, it traverses each triangular facet in the 3D mesh model, and calculates two edge vectors based on the coordinates of the three vertices of the facet: the difference between the coordinates of the second and first vertices is one edge vector; the difference between the coordinates of the third and first vertices is another edge vector. The two edge vectors are multiplied to obtain the normal vector of the triangular facet. This normal vector is then normalized to obtain the unit normal vector. The unit normal vector and the coordinates of any vertex are substituted into the plane point normal equation to calculate the plane equation coefficients.

[0071] Based on the coefficients of the plane equation, construct a quadratic error matrix for each triangular facet. For each vertex in the 3D mesh model of the drifting river, sum the quadratic error matrices of all adjacent triangular faces of the vertex; the summation result is the quadratic error matrix of the vertex. For each edge in the 3D mesh model of the drifting river, calculate the minimum geometric error (i.e., folding cost) incurred when folding the edge to a new vertex. The minimum geometric error is obtained by solving for the minimum quadratic form of the quadratic error matrix of the new vertex, expressed by the formula:

[0072] ;

[0073] in, This represents the minimum geometric error value produced after folding the edge. This represents the homogeneous coordinate vector of the new vertex after the edge is folded. Represents the homogeneous coordinate vector of the new vertex after edge folding. Transpose of;

[0074] The homogeneous coordinate vector of the new vertex after edge folding is obtained by decomposing the merged quadratic error matrix into blocks, resulting in a 3... A symmetric submatrix of 3, a 3 A column vector of 1 and a scalar constant; by solving a system of linear equations with symmetric submatrices as coefficient matrices and negative column vectors as constant terms, the optimal coordinates of the new vertex in three-dimensional space are obtained. The optimal coordinates are then combined with the homogeneous constant 1 to form a homogeneous coordinate vector.

[0075] The folding cost is determined by both edge length and local curvature change. Each iteration selects the edge with the minimum folding cost for folding. The folding operation merges the two vertices of the edge and removes associated triangular faces, while updating the geometric information of adjacent faces. The edge folding simplification algorithm iterates until the number of triangular faces reaches a preset target (10,000–100,000), generating a simplified 3D mesh model of the rafting river. On the surface of the simplified 3D mesh model, specific triangular face clusters are selected according to game design requirements and marked as virtual interaction areas. Simultaneously, based on the rafting river channel boundary and obstacle positions, feasible region boundaries are delineated on the surface of the simplified 3D mesh model. The 3D mesh model of the rafting river with virtual interaction area markings and feasible region boundaries is defined as the rafting river virtual scene model.

[0076] It should also be noted that the preset target (10,000 to 100,000) is set according to the trade-off between accuracy and performance in the application scenario; when the preset target is less than 10,000, it will lead to the loss of geometric accuracy, which will directly affect the game function and experience; when the preset target is greater than 100,000, it will put unbearable pressure on the calculation and cannot guarantee the smooth operation of the game.

[0077] S1.4 It should be noted that a photometric sensor array is deployed along the banks of the rafting river to continuously measure the ambient light intensity, principal direction vector of the light source, and color temperature of the rafting river using a high-frequency sampling method, thereby obtaining ambient light data of the rafting river; at the same time, an optical tracking camera deployed above the rafting river captures the spatial position and orientation of optical markers on the interactive device held by the tourist, generating spatial pose data of the interactive device; the triggering component of the interactive device generates an electronic signal when the tourist operates it, generating an interactive trigger signal; the spatial pose data of the interactive device and the interactive trigger signal together constitute the interaction data between the tourist and the interactive device.

[0078] S2. Based on the interaction data between tourists and interactive devices, identify the tourists' interactive actions and determine the occurrence of interactive events. When an interactive event is determined to have occurred, generate a virtual interactive ray starting from the position of the interactive device.

[0079] S2.1 It should be noted that, based on the interaction data between the tourist and the interactive device, the status of the interaction trigger signal is continuously monitored. When the interaction trigger signal is detected to change from an invalid state to an effective state, the occurrence of an interaction event is determined. After the interaction event occurs, the spatial pose data of the interactive device (including position coordinates and orientation angle) is read.

[0080] S2.2 It should be noted that the position coordinates in the spatial pose data are taken as the origin of the virtual interaction ray, and the direction of the directional angle is taken as the direction of the virtual interaction ray. In the global coordinate system of the rafting river virtual scene model, the virtual interaction ray is instantiated using the ray equation, where the ray equation is expressed by the formula:

[0081] ;

[0082] in, This represents the instantaneous three-dimensional spatial coordinate vector of any point on the virtual interaction ray. Represents the origin of the virtual interaction ray. Represents a scalar parameter along the direction of the virtual interaction ray. Represents the direction vector of the virtual interactive ray;

[0083] S3. Perform collision detection between the virtual interactive ray and the virtual interactive objects in the rafting river virtual scene model. When a collision is detected, record the coordinate information of the collision point.

[0084] S3.1 Perform collision detection between the virtual interactive ray and the virtual interactive objects in the rafting river virtual scene model;

[0085] It should be noted that the parametric equation of the virtual interaction ray is intersected with the triangular facet of each virtual interaction object in the rafting river virtual scene model. That is, for each triangular facet of each virtual interaction object, the intersection parameters of the virtual interaction ray and the plane containing the triangular facet are solved, which can be expressed by the formula:

[0086] ;

[0087] in, Indicates the intersection parameters. This represents the direction vector of the virtual interactive ray. This represents the normal vector of the triangular facet. Represents the origin of the virtual interaction ray. This represents the coordinate vector of the first vertex of the triangular facet.

[0088] Determine whether the intersection point parameters are within the valid range and whether the intersection point is within the boundary of the triangular facet. If the intersection point satisfies both the valid range (in meters) and the boundary of the triangular facet, record the intersection point parameters and the identifier of the virtual interactive object to which this intersection point belongs. After traversing all triangular facets of all virtual interactive objects, select the intersection point with the smallest orthogonal intersection point parameter from all recorded intersection points. This intersection point is the first collision between the virtual interactive ray and the drifting river virtual scene model. At the same time, record the coordinate information of the first collision intersection point.

[0089] It should also be noted that the effective range is [0.5, 30.0], which is set after a strict trade-off between physical rationality, gameplay design and real-time computing performance. When the effective range is less than 0.5, it will be exposed to the risk of physical errors, leading to unpredictable logical errors. When the effective range is greater than 30.0, it will introduce a lot of meaningless calculations, seriously sacrificing the smoothness of the game and violating the original design intention of the game.

[0090] S4. Based on the ambient light data, call the graphics rendering tool to render the virtual interactive objects that collide in real time, and calculate the pixel-level color and brightness values ​​of the virtual interactive objects to generate target holographic image frames.

[0091] S4.1 It should be noted that, based on the principal direction vector of the light source, the diffuse reflection illumination component of each pixel on the surface of the virtual interactive object is calculated, expressed by the formula:

[0092] ;

[0093] in, Indicates the diffuse light component. This represents the diffuse reflectance coefficient of the surface of a virtual interactive object. This represents the surface normal vector of the virtual interactive object at the point of collision. Represents the principal direction vector of the light source. Indicates the luminous intensity of the light source;

[0094] The proportions of the red, green, and blue channels of the diffuse light component are adjusted according to the color temperature to achieve color temperature adaptation of the spectral energy distribution. Ambient occlusion calculation is performed on the virtual interactive object. This calculation involves emitting a large number of sampling rays from each point on the surface of the virtual interactive object into the hemispherical space, while simultaneously detecting the occlusion between the sampling rays and the geometry of the virtual interactive object itself. Based on the occlusion situation, a two-dimensional texture map is baked for each point, which is the ambient occlusion map. During the rendering stage, a texture sampling operation is performed on the ambient occlusion map based on the texture coordinates of each pixel on the surface of the virtual interactive object. The texture sampling operation uses a bilinear interpolation algorithm to find the corresponding texel value in the ambient occlusion map based on the texture coordinates of the pixel. This texel value is the ambient occlusion coefficient corresponding to the pixel.

[0095] The final ambient light component is calculated based on the ambient light shading coefficient, and expressed by the formula:

[0096] T;

[0097] in, Indicates the final ambient light component. The base ambient light component is represented by the product of the light intensity and color temperature of the drifting river, and T represents the ambient light shading coefficient.

[0098] The half-angle vector is calculated based on the principal direction vector of the light source and the view direction vector (the unit vector pointing from a point on the surface of the virtual interactive object to the position of the virtual camera), expressed by the formula:

[0099] ;

[0100] in, Represents a half-angle vector. Represents the principal direction vector of the light source. A unit vector representing a point on the surface of a virtual interactive object pointing to the position of the virtual camera;

[0101] The probability density of the consistency between the direction of the surface normal vector and the half-angle vector of the virtual interactive object at the collision point is calculated using the normal distribution function. The normal distribution function adopts the GGX model, which can be expressed as:

[0102] ;

[0103] in, Represents the value of the normal distribution function. Represents pi (π). Represents the roughness coefficient. This represents the surface normal vector of the virtual interactive object at the point of collision. Represents a half-angle vector;

[0104] The roughness coefficient is preset based on the material of the virtual interactive object, and its value ranges from 0.0 to 1.0. When the roughness coefficient is less than 0.0, it will not make the virtual interactive object more "smooth" but will only cause rendering failure. When the roughness coefficient is greater than 1.0, it will not make the virtual interactive object more "rough" but will only cause the material of the virtual interactive object to become dark and distorted.

[0105] The geometric occlusion function is used to calculate the occlusion and shadow effects of light on the surface microstructure of virtual interactive objects. The geometric occlusion function adopts the Smith model and is expressed by the formula:

[0106] ;

[0107] in, This represents the geometric occlusion function value (the combined occlusion and shadow effect of the surface microstructure of the virtual interactive object on light). This represents the surface normal vector of the virtual interactive object at the point of collision. This represents a unit vector pointing from a point on the surface of a virtual interactive object to the position of the virtual camera. Represents the direction vector of light rays;

[0108] The Fresnel equation is used to calculate the reflectivity of light at a specific incident angle on the surface of a virtual interactive object. The Fresnel equation is solved based on the angle between the view direction vector and the half-angle vector, and can be expressed as:

[0109]

[0110] in, This represents the Fresnel reflectance value (the proportion of light reflected at a specific angle of incidence on the surface of a virtual interactive object). This represents the base reflectivity of the material of a virtual interactive object under perpendicular incidence. This represents a unit vector pointing from a point on the surface of a virtual interactive object to the position of the virtual camera. Represents a half-angle vector;

[0111] The specular reflection component is obtained by multiplying the calculated results of the normal distribution function, geometric occlusion function, and Fresnel equation, and then multiplying this product with the specular reflection coefficient of the virtual interactive object's material. The specular reflection coefficient is a preset physical parameter based on the virtual interactive object's material type, used to control the intensity and color of specular reflection. Its value ranges from 0.0 to 1.0. When the specular reflection coefficient is less than 0.0, it will not produce any realistic effect, only causing rendering abnormalities, and a specular reflection coefficient less than 0 is physically impossible. When the specular reflection coefficient is greater than 1.0, it will lead to severe overexposure and distortion, completely destroying the immersive experience.

[0112] The diffuse light component, the final ambient light component, and the specular reflection component are added together to obtain the final pixel-level color and brightness value (RGB vector) of each pixel on the surface of the virtual interactive object. The pixel-level color and brightness value of all pixels together constitute the target holographic image frame.

[0113] It should also be noted that the surface normal vector of the virtual interactive object at the collision point is collected from the collision point coordinate information. That is, based on the collision point coordinate information, the triangular facet where the collision point is located in the rafting river virtual scene model is located, and the interpolated normal vector at the collision point (i.e., the surface normal vector of the virtual interactive object at the collision point) is calculated using the coordinates of the three vertices of the triangular facet. The GGX model and the Smith model are physically based analytical models, with mathematical formulas at their core, and are not machine learning models that learn parameters through data.

[0114] S5. Calculate the mechanical effects when the virtual interactive ray collides with the virtual interactive object based on the collision point coordinate information;

[0115] S5.1. Collect the normal vector of the surface of the virtual interactive object at the collision point from the collision point coordinate information, and calculate the impulse generated by the collision in combination with the impulse theorem.

[0116] It should be noted that, based on the collision point coordinates, the triangular facet containing the collision point is located in the virtual scene model of the drifting river. After determining the surface normal vector of the virtual interactive object at the collision point according to the geometric definition of the triangular facet, the impulse generated by the collision is calculated using the impulse theorem, expressed by the formula:

[0117] ;

[0118] in, This represents the impulse generated by the collision. This represents the elasticity coefficient of the material of a virtual interactive object. This represents the surface normal vector of the virtual interactive object at the point of collision. A virtual mass scalar representing a virtual interactive ray;

[0119] S5.2 Calculate the changes in linear velocity and angular velocity of virtual interactive objects using the impulse generated by the collision to obtain the mechanical effects;

[0120] Furthermore, the linear momentum change is obtained by multiplying the impulse generated by the collision with the surface normal vector of the virtual interactive object at the collision point; the linear velocity change of the virtual interactive object is obtained by quotienting the linear momentum change with the mass of the virtual interactive object; simultaneously, the torque is obtained by multiplying the linear momentum change with the center of mass of the virtual interactive object, which is equal to the cross product of the collision point position vector and the impulse; the angular acceleration change of the virtual interactive object is obtained by quotienting the torque with the moment of inertia of the virtual interactive object, and the angular velocity change of the virtual interactive object is obtained by integrating the angular acceleration change over time; the linear velocity change and the angular velocity change of the virtual interactive object together constitute the mechanical effect describing the change in the motion state of the virtual interactive object.

[0121] It should be noted that the mass of a virtual interactive object is a preset scalar parameter directly assigned by the virtual interactive object's attribute definition, with a value ranging from 0.1kg to 100,000kg, and the value is based on the balance between physical rationality and game design goals.

[0122] The centroid of a virtual interactive object is a spatial point determined by the object's own geometry and world coordinates. It is calculated by summing the vertex coordinates of all triangular facets in the 3D mesh model of the drifting river and then quotienting the sum with the total number of vertices.

[0123] The moment of inertia of a virtual interactive object is calculated using the formula for moment of inertia, based on its mass, geometry, and center of mass position. The formula is as follows:

[0124] ;

[0125] in, Indicates the quality of virtual interactive objects. Indicates the height of the rafting river channel. Indicates the length of the rafting river channel. Indicates the width of the river channel used for rafting. This represents the moment of inertia of a virtual interactive object.

[0126] S6. Map the mechanical effects to physical disturbance parameters of the rafting vehicle and generate physical feedback control commands.

[0127] S6.1 It should be noted that, based on the linear velocity and angular velocity changes of the virtual interactive object in the mechanical effects, the physical disturbances (i.e., physical disturbance parameters, including thrust vector and torque vector) that the rafting vehicle should experience in the virtual water body are calculated. The thrust vector is the product of the thrust conversion coefficient and the linear velocity change of the virtual interactive object; the torque vector is the product of the torque conversion coefficient and the angular velocity change of the virtual interactive object. The magnitude of the thrust vector is proportional to the linear velocity change of the virtual interactive object, and its direction is opposite to the direction of motion of the virtual interactive object. The magnitude of the torque vector is proportional to the angular velocity change of the virtual interactive object, and its direction is determined by the direction of the angular velocity change.

[0128] The physical disturbance parameters are encoded into physical feedback control commands, which include fields for target underwater thruster number, thrust magnitude, torque magnitude, direction of action, and duration.

[0129] It should also be noted that the thrust conversion factor ranges from 0.1 to 10.0, and is set based on game design, physical simulation scale matching, and user experience. When the thrust conversion factor is less than 0.1, the physical feedback generated by virtual interaction will be so weak as to be imperceptible, thus destroying the sense of immersion. When the thrust conversion factor is greater than 10.0, slight virtual interaction will be amplified into a violent physical impact, bringing a series of safety issues.

[0130] The torque conversion coefficient ranges from 0.01 to 5.0, and is set based on human comfort and safety. When the torque conversion coefficient is less than 0.01, the dynamic effect of the virtual interactive object's rotation cannot be effectively transmitted to the physical world, resulting in the destruction of the dynamic immersion. When the torque conversion coefficient is greater than 5.0, slight virtual rotational interaction will be amplified into violent and dangerous physical rotational impact, causing severe dizziness and nausea in tourists, which is a situation that must be avoided.

[0131] S7. Send the physical feedback control command to the physical feedback device to drive the physical feedback device to generate physical environmental disturbances synchronized with the target holographic image frame and apply them to the rafting river recreation vehicle.

[0132] S7.1 It should be noted that the physical feedback control command is sent to the physical feedback device (underwater thruster array) through a wireless communication link. After receiving and parsing the physical feedback control command, the physical feedback device performs corresponding actions according to the thrust magnitude, thrust direction and torque magnitude in the physical environment disturbance parameters. That is, the underwater thruster array sprays water according to the physical environment disturbance parameters and forms a specific wave sequence in the river channel. When the wave sequence propagates to the bottom of the rafting vehicle as a physical environment disturbance, the holographic projection device immediately displays the target holographic image frame.

[0133] The wave sequence generates upward buoyancy and lateral wave force on the hull of the rafting vehicle. The combined effect of buoyancy and wave force causes the rafting vehicle to produce synchronous displacement and swaying. This displacement and swaying action corresponds perfectly in time and logic to the visual motion effect of the virtual interactive object in the target holographic image frame.

[0134] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0135] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the real-time feedback method for implementing a holographic image-based interactive drifting river game system as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0136] In summary, this invention achieves seamless integration of holographic images with real lighting environments by collecting ambient light data and combining it with a physically based rendering method. Simultaneously, it calculates mechanical effects, establishes the physical causal logic of interactive behavior, maps virtual mechanical parameters into controllable physical perturbation commands, and realizes precise conversion and safe adaptation of force feedback between the virtual and real worlds. Through spatiotemporal synchronization between physical feedback devices and holographic images, it constructs an immersive closed-loop experience with highly consistent visual, auditory, and tactile sensations, thereby achieving a comprehensive entertainment effect of deep integration of virtual and real worlds, realistic and credible interaction, and a coherent and immersive experience.

[0137] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A real-time feedback method for a holographic image-interactive river drifting game system, characterized in that: include, Collect 3D point cloud data of the rafting river, and use surface reconstruction algorithm to model the 3D point cloud data of the rafting river into a virtual scene model of the rafting river. At the same time, collect ambient light data on the rafting river and interaction data between tourists and interactive devices. Based on the interaction data between tourists and interactive devices, the system identifies tourists' interactive actions and determines the occurrence of interactive events. When an interactive event is determined to have occurred, a virtual interactive ray is generated starting from the position of the interactive device. Collision detection is performed between the virtual interactive ray and the virtual interactive objects in the rafting river virtual scene model. When a collision is detected, the coordinate information of the collision point is recorded. Based on ambient light data, the graphics rendering tool is invoked to render the virtual interactive objects that collide in real time, and the pixel-level color and brightness values ​​of the virtual interactive objects are calculated to generate target holographic image frames. Based on the collision point coordinates, the mechanical effects when the virtual interactive ray collides with the virtual interactive object are calculated, specifically: The normal vector of the virtual interactive object's surface at the collision point is collected from the collision point coordinate information, and the impulse generated by the collision is calculated using the impulse theorem. Specifically: Locate the triangular facet containing the collision point in the virtual scene model of the drifting river based on the collision point coordinates. After determining the surface normal vector of the virtual interactive object at the collision point based on the geometric definition of the triangular facet where the collision point is located, the impulse generated by the collision is calculated by applying the impulse theorem. The impulse generated by the collision is used to calculate the linear and angular velocity changes of the virtual interactive object, thereby obtaining the mechanical effects, specifically: The linear momentum change is obtained by multiplying the impulse generated by the collision with the surface normal vector of the virtual interactive object at the collision point. The linear momentum change is then divided by the mass of the virtual interactive object to obtain the linear velocity change of the virtual interactive object. At the same time, the torque is equal to the cross product of the collision point position vector and the impulse. The quotient of the torque and the moment of inertia of the virtual interactive object is used to obtain the change of angular acceleration of the virtual interactive object. The change of angular acceleration is then integrated over time to obtain the change of angular velocity of the virtual interactive object. The linear velocity change and angular velocity change of a virtual interactive object together constitute the mechanical effect describing the change in the motion state of the virtual interactive object; The mechanical effects are mapped to physical disturbance parameters of the rafting vehicle, and physical feedback control commands are generated. The physical feedback control command is sent to the physical feedback device, which drives the physical feedback device to generate physical environmental disturbances synchronized with the target holographic image frame, and then acts on the rafting vehicle.

2. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 1, characterized in that: The process involves collecting 3D point cloud data of the rafting river and using a surface reconstruction algorithm to model the 3D point cloud data into a virtual scene model of the rafting river. Specifically: During the initialization phase of the rafting river game, a 3D laser scanner was used to scan the real rafting river channel at multiple stations to obtain 3D point cloud data of the rafting river, and the 3D point cloud data of the rafting river was then registered. The registered 3D point cloud data of the drifting river was reconstructed into a 3D mesh model of the drifting river using the Poisson surface reconstruction algorithm. The 3D mesh model of the rafting river is simplified by reducing the number of triangular faces while maintaining geometric features. The boundaries of the virtual interaction area and the feasible area are marked on the simplified 3D mesh model of the rafting river to obtain the virtual scene model of the rafting river.

3. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 2, characterized in that: The ambient light data on the drifting river includes light intensity, principal direction vector of the light source, and color temperature; The interaction data between the tourist and the interactive device includes the spatial pose data of the interactive device and the interaction trigger signal.

4. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 3, characterized in that: The system identifies tourist interaction actions and determines the occurrence of interaction events based on interaction data between tourists and interactive devices. When an interaction event is determined to have occurred, a virtual interaction ray is generated starting from the position of the interactive device. Specifically: Based on the interaction data between tourists and interactive devices, the status of interaction trigger signals is continuously monitored. When the interaction trigger signal is detected to change from an invalid state to an valid state, it is determined that an interactive event has occurred. After an interactive event occurs, read the spatial pose data of the interactive device; The spatial pose data of the interactive device includes position coordinates and orientation angle; Using the position coordinates as the origin of the virtual interaction ray and the direction of the angle as the direction of the virtual interaction ray, a virtual interaction ray is instantiated and generated in the virtual scene model of the drifting river.

5. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 4, characterized in that: The process involves performing collision detection between the virtual interactive ray and virtual interactive objects in the virtual scene model of the drifting river. When a collision is detected, the coordinates of the collision point are recorded. Specifically: Collision detection is performed between the virtual interactive ray and the virtual interactive objects in the virtual scene model of the drifting river. During collision detection, the mathematical equation of the virtual interaction ray is calculated and its spatial positional relationship with the triangular facet geometry of the virtual interaction object is determined. When the calculation results show that the virtual interaction ray and the triangular facet geometry of the virtual interaction object intersect, a collision is determined to have been detected. After a collision occurs, calculate the coordinates of the intersection point between the virtual interactive ray and the triangular facet geometry of the virtual interactive object, and record the intersection point coordinates as the collision point coordinate information.

6. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 5, characterized in that: The process involves using ambient light data to call a graphics rendering tool to render the colliding virtual interactive objects in real time, calculating the pixel-level color and brightness values ​​of the virtual interactive objects, and generating a target holographic image frame. Specifically: Calculate the diffuse reflection light component of each pixel on the surface of the virtual interactive object based on the principal direction vector of the light source; Based on the diffuse illumination component, the spectral energy distribution of the pixels is adjusted using color temperature. Then, the ambient light occlusion and specular reflection components are integrated using a graphics rendering tool. Finally, the pixel-level color and brightness values ​​of the virtual interactive object surface are calculated pixel by pixel to obtain the target holographic image frame.

7. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 6, characterized in that: The mapping of mechanical effects to physical disturbance parameters of the rafting vehicle refers to calculating the physical disturbance parameters that the rafting vehicle should experience in the virtual water body based on the linear velocity and angular velocity changes of the virtual interactive object in the mechanical effects.

8. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 7, characterized in that: The generation of physical feedback control commands refers to encoding physical disturbance parameters into commands that can be recognized and executed by the physical feedback device.

9. The real-time feedback method of the holographic image interaction-based river drifting game system as described in claim 8, characterized in that: The process of sending physical feedback control commands to a physical feedback device, driving the device to generate physical environmental disturbances synchronized with the target holographic image frame, and applying these disturbances to the rafting vehicle, specifically involves: The physical feedback control command is sent to the physical feedback device via a wireless communication link; After receiving and parsing the physical feedback control command, the physical feedback device generates a wave sequence in the river. After synchronizing the target holographic image frame with the wave sequence, it generates thrust and swaying force on the rafting vehicle.

Images