Exhibition hall immersive multimedia content interaction system based on naked eye 3D

Abstract

The invention relates to the technical field of virtual reality and immersive multimedia display, and discloses an exhibition hall immersive multimedia content interaction system based on naked eye 3D, which comprises a virtual-real space calibration module, a double-view cone mixed rendering module, a bidirectional inverse projection illumination processing module and an interaction synthesis module. According to the system, physical and virtual space mapping is established, a static wide-angle rendering channel based on a fixed viewpoint and a dynamic off-axis rendering channel based on an observer viewpoint are constructed in parallel, a two-way inverse projection illumination processing module calculates a parallax angle and constructs an inverse view angle compensation matrix, illumination of the static channel is corrected to be matched with physical environment light, and the dynamic off-axis rendering channel is constructed. And the interactive synthesis module generates a shadow mask through a phantom geometry, and realizes cross-channel light and shadow interactive fusion by using Gaussian deformation and normal redirection technologies. According to the method, the problems of perspective mismatch and light and shadow inconsistency of the foreground and the background in layered rendering are effectively solved, and the sense of reality of immersive interaction is improved.

Description

Technical Field

[0001] This invention relates to the field of virtual reality and immersive multimedia display technology, specifically to an immersive multimedia content interaction system for exhibition halls based on naked-eye 3D. Background Technology

[0002] In large multimedia exhibition halls or immersive experience spaces, multi-channel projection or LED displays are often used to construct CAVE display systems. In order to present a naked-eye 3D visual effect with a sense of depth, existing technologies usually combine motion capture equipment to obtain the observer's position and use off-axis projection technology of asymmetric visual cones to reconstruct the three-dimensional scene in real time.

[0003] However, in large-screen, high-resolution display environments, real-time dynamic reconstruction of the entire scene places extremely high demands on the performance of graphics computing hardware. Furthermore, when the observer moves, if the background image in a large field of view undergoes significant perspective changes with the viewpoint, it can easily cause dizziness and other discomfort. Therefore, existing technical solutions often employ a layered rendering strategy, where the background layer uses pre-rendered video or static images from a fixed viewpoint, while only the interactive objects in the foreground are dynamically rendered in real-time following the viewpoint.

[0004] While this layered approach of a fixed background and a dynamic foreground reduces computational load and maintains visual stability of the background, it leads to inconsistencies in the spatial geometry of the foreground and background. Because the vanishing point of the background image is fixed, while the perspective of the foreground objects changes with the observer's viewpoint, a perspective mismatch occurs in space, causing the foreground objects to appear visually shifted relative to the background. Furthermore, exhibition halls typically have fixed physical light sources. When the observer's position changes, differences in lighting direction and intensity arise between the dynamically calculated foreground based on physical lighting and the statically lit background. Moreover, the dynamic foreground struggles to project shadows that match the current viewpoint's perspective onto a static background that only possesses color information, resulting in a reduced integration of the virtual scene with the physical environment. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an immersive multimedia content interaction system for exhibition halls based on naked-eye 3D, which solves the technical problems of perspective mismatch between foreground and background, inconsistent light and shadow directions, and lack of interactive occlusion relationships caused by the movement of the observer's viewpoint when existing large-field-of-view immersive display systems use layered rendering.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: an immersive multimedia content interaction system for exhibition halls based on naked-eye 3D, comprising: an immersive display terminal, a multi-dimensional perception sensor group, and a graphics computing workstation; the graphics computing workstation includes a virtual-real space calibration module, a dual-view frustum hybrid rendering module, a bidirectional inverse projection lighting processing module, and an interactive synthesis module;

[0007] The virtual-real space calibration module is configured to establish a mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system, and to convert the observer's viewpoint coordinates and physical light source vectors collected by the multi-dimensional perception sensor group into parameters under the virtual rendering scene coordinate system.

[0008] The dual-view frustum hybrid rendering module is configured to construct a static wide-angle rendering channel and a dynamic off-axis rendering channel in parallel; wherein, the static wide-angle rendering channel renders the environment background object based on a preset fixed viewing point, and the dynamic off-axis rendering channel renders the interactive foreground object based on the observer's viewpoint coordinates;

[0009] The bidirectional reverse projection lighting processing module is configured to calculate, based on the physical light source vector, the first lighting transformation parameters acting on the static wide-angle rendering channel and the second lighting transformation parameters acting on the dynamic off-axis rendering channel, respectively.

[0010] The interactive compositing module is configured to fuse the background image generated by the static wide-angle rendering channel with the foreground image generated by the dynamic off-axis rendering channel based on the observer's viewpoint coordinates and the collision state of the interactive foreground object, and then output the fusion to the immersive display terminal.

[0011] Preferably, the dual-view frustum hybrid rendering module performs off-axis projection calculations when constructing the dynamic off-axis rendering channel:

[0012] Obtain the corner coordinates of the screen plane of the immersive display terminal in the virtual rendering scene coordinate system, and construct the basis vector of the screen plane;

[0013] Project the observer's viewpoint coordinates onto the screen space defined by the basis vectors, and calculate the vertical distance and relative offset of the observer's viewpoint coordinates relative to the screen plane;

[0014] Based on the vertical distance and relative offset, the left and right boundary values ​​and the top and bottom boundary values ​​of the asymmetric view frustum are determined, and then an off-axis projection matrix is ​​generated for rendering the interactive foreground object.

[0015] Preferably, the dual-view frustum hybrid rendering module also loads phantom geometry in the dynamic off-axis rendering channel;

[0016] The phantom geometry is a geometric copy of the environment background object. The phantom geometry is configured as an invisible material and only participates in depth testing and shadow calculation.

[0017] The dynamic off-axis rendering channel uses the phantom geometry to receive the shadows cast by the interactive foreground object and generates a shadow mask containing only shadow information.

[0018] The interactive compositing module uses the shadow mask to modulate the brightness of the background image generated by the static wide-angle rendering channel.

[0019] Preferably, the bidirectional inverse projection illumination processing module calculates the second illumination transformation parameters in the following way:

[0020] Obtain the physical light source reference direction vector after conversion by the virtual-real space calibration module;

[0021] The physical light source reference direction vector is directly mapped to the light source incident direction in the dynamic off-axis rendering channel, so that the lighting direction of the interactive foreground object is consistent with the actual lighting direction of the physical exhibition hall.

[0022] Preferably, the bidirectional inverse projection illumination processing module calculates the first illumination transformation parameters in the following way:

[0023] Calculate the parallax rotation angle between the dynamic line-of-sight vector corresponding to the observer's viewpoint coordinates and the static line-of-sight vector corresponding to the fixed observation viewpoint, with the scene center as the reference point.

[0024] Construct an inverse viewpoint compensation matrix based on the parallax rotation angle;

[0025] The physical light source reference direction vector is rotated inversely using the inverse perspective compensation matrix to generate a corrected lighting vector, which is then used as the light source incident direction of the static wide-angle rendering channel.

[0026] Preferably, the interactive synthesis module is further configured to perform mesh deformation calculations:

[0027] When the interactive node corresponding to the observer's viewpoint coordinates is detected to enter the collision bounding box of the interactive foreground object, the distance between the interactive node and the vertex of the interactive foreground object surface is calculated.

[0028] The vertex displacement is calculated using the Gaussian radial basis function, and the vertex displacement decreases non-linearly with the increase of the distance.

[0029] The vertex displacement is applied during the vertex shader stage to modify the mesh geometry of the interactive foreground object in real time.

[0030] Preferably, the interactive composition module is further configured to perform cross-channel normal redirection:

[0031] Calculate the tangent space normal perturbation vector of the interactive foreground object caused by mesh deformation;

[0032] Write the normal perturbation vector into the shared normal perturbation texture;

[0033] When rendering the environment background object, the static wide-angle rendering channel samples the normal perturbation texture and superimposes the sampled perturbation vector onto the original normal of the environment background object, thereby changing the surface lighting and reflection properties of the environment background object.

[0034] Preferably, the specific method by which the interactive synthesis module fuses images is as follows:

[0035] Obtain the color value of the background image, the color value of the foreground image, and the grayscale value of the shadow mask;

[0036] The grayscale values ​​of the shadow mask are used to perform a weighted multiplication of the color values ​​of the background image to obtain a darkened background image;

[0037] The color values ​​of the foreground image are superimposed onto the darkened background image to obtain a composite image;

[0038] The screen geometry correction lookup table is invoked to perform two-dimensional nonlinear resampling on the synthesized image, generating a final output frame that adapts to the physical curvature of the immersive display terminal.

[0039] Preferably, the virtual-real space calibration module stores a rigid body transformation calibration matrix;

[0040] The virtual-real space calibration module transforms the physical light source vector through the rotation component of the rigid body transformation calibration matrix, and transforms the observer's viewpoint coordinates through the rotation and translation components of the rigid body transformation calibration matrix.

[0041] Preferably, an immersive multimedia content interaction method for exhibition halls based on naked-eye 3D includes the following steps:

[0042] S1. Establish the mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system through the virtual and real space calibration module, and convert the collected observer viewpoint coordinates and physical light source vectors into parameters under the virtual rendering scene coordinate system.

[0043] S2. Construct a static wide-angle rendering channel and a dynamic off-axis rendering channel in parallel using a dual-view frustum hybrid rendering module;

[0044] S3. Based on the physical light source vector, the bidirectional reverse projection lighting processing module calculates the first lighting transformation parameters acting on the static wide-angle rendering channel and the second lighting transformation parameters acting on the dynamic off-axis rendering channel, respectively.

[0045] S4. Control the static wide-angle rendering channel to render the background image based on a fixed viewing point and the first lighting transformation parameters, and control the dynamic off-axis rendering channel to render the foreground image based on the observer's viewpoint and the second lighting transformation parameters.

[0046] S5. The background image and the foreground image are merged through the interactive synthesis module and output to the immersive display terminal.

[0047] This invention provides an immersive multimedia content interaction system for exhibition halls based on naked-eye 3D. It has the following beneficial effects:

[0048] 1. This invention adopts a dual-view frustum hybrid rendering architecture, which separates the rendering pipeline into a static wide-angle channel and a dynamic off-axis channel. The background part uses fixed viewpoint rendering to avoid severe perspective distortion of the background of the whole scene when the observer moves. The foreground part uses real-time off-axis projection to ensure the floating three-dimensional effect of interactive objects. This heterogeneous rendering method reduces the computational overhead of real-time reconstruction of the whole scene while maintaining the accuracy of naked-eye 3D vision, and takes into account the stability of the background and the three-dimensionality of the foreground interaction.

[0049] 2. This invention utilizes bidirectional inverse projection lighting matrix transformation technology to calculate the parallax angle between the observer's real-time viewpoint and the fixed viewpoint, and constructs an inverse perspective compensation matrix to correct the lighting vector of the static channel. This technology ensures that, even with a fixed background perspective, the direction of light and shadow on the background surface remains consistent with that of the dynamic foreground object illuminated by the physical light source, eliminating the light and shadow discontinuity caused by the inconsistency between the two rendering viewports and guaranteeing the matching of the lighting direction of the virtual content with the actual light source of the physical exhibition hall.

[0050] 3. This invention introduces a phantom geometry masking and cross-channel normal redirection mechanism. It uses invisible phantom geometry to receive the shadows cast by the foreground and generate a mask, solving the shadow occlusion problem across rendering channels. At the same time, it converts the foreground mesh deformation data based on Gaussian decay into normal perturbation texture and passes it to the background layer. This allows the interactive actions of the foreground to drive the static background to produce corresponding light refraction or surface ripples, improving the problem of lack of physical linkage between the foreground and background in traditional layered content and enhancing the realism of the interactive process. Attached Figure Description

[0051] Figure 1 This is a schematic diagram illustrating the system application scenario of the present invention;

[0052] Figure 2 This is a block diagram of the system functional modules of the present invention;

[0053] Figure 3 This is a flowchart of the method of the present invention.

[0054] Among them, 100 is the virtual-real space calibration module; 200 is the dual-view frustum hybrid rendering module; 300 is the bidirectional inverse projection lighting processing module; and 400 is the interactive compositing module. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example:

[0057] Please see the appendix Figure 1 -Appendix Figure 2 This invention provides an immersive multimedia content interaction system for exhibition halls based on naked-eye 3D, including an immersive display terminal, a multi-dimensional perception sensor group, and a graphics computing workstation.

[0058] Immersive display terminals are used to construct three-dimensional visual display spaces. Specifically, immersive display terminals take the form of L-shaped angled LED screens or curved LED screens. The display surface of the immersive display terminal is defined as the system's physical interactive interface, used to output the final synthesized naked-eye 3D video signal.

[0059] The multi-dimensional sensing sensor array is deployed within the physical exhibition space where the immersive display terminal is located. The multi-dimensional sensing sensor array is used to collect spatial position data, motion posture data, and ambient lighting data of the observer.

[0060] The graphics computing workstation communicates with the immersive display terminal and the multi-dimensional sensing sensor array. The workstation receives sensor data, performs rendering calculations, and sends the processed image data to the immersive display terminal.

[0061] The graphics computing workstation has internal control logic, which includes: a virtual and real space calibration module 100, a dual-view frustum blending rendering module 200, a bidirectional inverse projection lighting processing module 300, and an interactive compositing module 400.

[0062] The virtual-real space calibration module 100 is connected to the multi-dimensional sensing sensor group. The virtual-real space calibration module 100 is used to establish the mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system. The virtual-real space calibration module 100 converts the observer's head coordinates, hand end coordinates, and physical light source vectors collected by the multi-dimensional sensing sensor group into corresponding parameters in the virtual scene coordinate system.

[0063] The dual-view frustum hybrid rendering module 200 is connected to the virtual-real space calibration module 100. The dual-view frustum hybrid rendering module 200 is used to construct two independent rendering channels in parallel. The dual-view frustum hybrid rendering module 200 includes a static channel rendering unit and a dynamic channel rendering unit.

[0064] The static channel rendering unit generates a static background image containing the environmental background based on a preset fixed viewing point. The dynamic channel rendering unit generates a dynamic foreground image containing interactive objects and phantom geometry based on the real-time observer's viewpoint output by the virtual-real space calibration module 100 and using an off-axis projection algorithm.

[0065] The bidirectional inverse projection lighting processing module 300 is connected to the virtual-real space calibration module 100 and the dual-view frustum hybrid rendering module 200. The bidirectional inverse projection lighting processing module 300 is used to calculate the mapping transformation of physical light sources in virtual space.

[0066] The bidirectional inverse projection lighting processing module 300 calculates the first lighting transformation matrix acting on the static channel rendering unit and the second lighting transformation matrix acting on the dynamic channel rendering unit, based on the physical light source vector output by the virtual-real space calibration module 100 and the parallax angle between the real-time observer's viewpoint and the fixed observer's viewpoint. The dual-view frustum hybrid rendering module 200 performs shading calculations using the first and second lighting transformation matrices.

[0067] The interactive compositing module 400 is connected to the virtual-real space calibration module 100 and the dual-view frustum blending rendering module 200. The interactive compositing module 400 is used to handle spatial collision detection and image fusion.

[0068] The interactive compositing module 400 determines whether the hand's end-effector has made contact with a virtual interactive object based on the coordinates output by the virtual-real space calibration module 100. If contact occurs, the interactive compositing module 400 calculates mesh deformation data and updates it to the dual-view frustum blending rendering module 200. The interactive compositing module 400 then weights and overlays the static background image and the dynamic foreground image, performs geometric corrections for the physical form of the immersive display terminal, and finally generates the output video frame.

[0069] The overall working principle of the system is as follows: the multi-dimensional sensing sensor group collects data in real time and transmits it to the graphics computing workstation; the virtual and real space calibration module 100 performs spatial coordinate unification processing on the data.

[0070] Subsequently, the bidirectional inverse projection lighting processing module 300 calculates the lighting matrix, and the dual-view frustum hybrid rendering module 200 generates a background image with fixed perspective and a foreground image with dynamic perspective based on the unified coordinate data and the calculated lighting matrix.

[0071] Finally, the interactive synthesis module 400 merges the two images based on depth occlusion and interactive deformation state, and presents naked-eye 3D interactive content with physical light and shadow consistency through an immersive display terminal.

[0072] Please see the appendix Figure 1 -Appendix Figure 2 The virtual-physical space calibration module 100 is configured to perform geometric alignment between physical space and virtual space, which specifically includes the following sub-steps:

[0073] S101, Define the coordinate system.

[0074] To achieve precise alignment between physical interaction and virtual content, the system predefines four basic coordinate systems:

[0075] Physical world coordinate system W phys A right-handed rectangular coordinate system is established with the center point of the physical ground in front of the immersive display terminal as the origin, the direction perpendicular to the ground as the Z-axis, and the direction parallel to the bottom edge of the screen as the X-axis.

[0076] Virtual scene coordinate system W virt The three-dimensional world coordinate system inside the graphics rendering engine, used to place virtual objects and calculate lighting;

[0077] Screen space coordinate system W scr : A two-dimensional coordinate system defined on the physical surface of the immersive display terminal, used for the final image mapping;

[0078] Camera observation coordinate system W view : A local coordinate system with the optical center of the virtual camera as the origin.

[0079] S102, collects and preprocesses raw sensor data.

[0080] The virtual-real space calibration module 100 reads data from the multi-dimensional sensing sensor group via an interface. For acquiring observer position data, the multi-dimensional sensing sensor group specifically employs a LiDAR array or a multi-view stereo vision camera. The sensor output data is then processed by Kalman filtering or particle filtering for noise reduction to generate the observer's head position in the physical world coordinate system W. physcoordinates below and the coordinates of the end node of the hand The specific implementation algorithm of Kalman filtering is a well-known technique in the field of signal processing, and will not be elaborated here.

[0081] For acquiring physical environment lighting data, a multi-dimensional sensing sensor group uses a lighting sensor array deployed on the ceiling of the exhibition hall to output the direction vector of the main physical light source. and the scalar of light intensity I phys .

[0082] S103, construct the rigid body transformation calibration matrix.

[0083] The virtual-real space calibration module 100 stores a pre-calculated calibration matrix M. calib This matrix describes the coordinates from the physical world coordinate system W. phys To the virtual scene coordinate system W virt The rigid body transformation relationship. M calib Let be a 4×4 homogeneous transformation matrix, defined as follows:

[0084]

[0085] Where, r ij (i,j=1,2,3) constitute the 3×3 rotation matrix components R calib , used to align the axial directions of physical space and virtual space; t x ,t y ,t z This constitutes the translation vector component T calib This matrix is ​​used to align the origin positions of two spaces. It is obtained by pre-placing calibration references at specific locations in the physical space and setting anchor points at corresponding locations in the virtual space, and then solving it using the least squares method.

[0086] S104, Perform spatial coordinate mapping transformation.

[0087] Based on the above calibration matrix, the virtual-real space calibration module 100 converts physical coordinates into virtual coordinates in real time. The calculation formula for the observer's viewpoint position is:

[0088] P user =M calib ·[x p ,y p ,z p ,1] T ;

[0089] Among them, P user These are the viewpoint coordinates of the observer in the virtual scene, used to subsequently drive the camera position in the dynamic rendering pass.

[0090] The calculation formula for the hand interaction node position is:

[0091] P inter =M calib ·[x h ,y h ,z h ,1] T ;

[0092] Among them, P inter These are the coordinates of interactive touchpoints in the virtual scene, used for subsequent collision detection and mesh deformation calculation.

[0093] For the direction vector of the physical light source, since the direction vector is not affected by translation, only the rotation matrix components are applied for transformation:

[0094]

[0095] in, This is the reference direction vector of the physical light source in the virtual scene coordinate system. It should be noted that this reference vector... It is not used directly for the final coloring, but rather as the input base data for the subsequent bidirectional inverse projection lighting processing module 300.

[0096] S105 defines a virtual mapping of the screen's physical boundaries.

[0097] To support subsequent off-axis projection calculations, the virtual-real space calibration module 100 also needs to be in the virtual scene coordinate system W. virt The system precisely defines the physical edges of the immersive display terminal. Specifically, the system reads the physical size parameters of the immersive display terminal and combines them with the calibration matrix M. calib The coordinates of three key corner points of the screen plane in virtual space were calculated: the coordinates of the lower left corner of the screen, V. bl The coordinates V at the bottom right corner of the screen br and the coordinates V of the top left corner of the screen tl These three coordinate points define the projection plane in the virtual space, serving as the geometric reference for the dynamic view frustum construction in the subsequent dual-view frustum hybrid rendering module 200.

[0098] Please see the appendix Figure 1 -Appendix Figure 2 The dual-view frustum blending rendering module 200 is configured to process two independent rendering flows from different perspectives in parallel to achieve spatial fusion of static background and dynamic foreground. This process specifically includes the following sub-steps:

[0099] S201, configures a dual-channel scene layer structure.

[0100] The dual-view frustum blending rendering module 200 initializes two independent frame buffers in graphics memory: a static frame buffer FB.stat and dynamic frame buffer FB dyn At the same time, three types of geometric objects are defined in the scene graph:

[0101] The first category is the environmental background object G. bg This includes static models such as distant terrain and vegetation, and the object is only marked as visible in the static channel.

[0102] The second category is interactive foreground objects G. dyn This includes dynamic models such as virtual creature characters and particle effects, and the object is only marked as visible in the dynamic channel;

[0103] The third category is phantom geometry G. phan This object is the environment background object G. bg A low-precision copy of the geometry. Phantom geometry G phan It is only marked as visible in the dynamic channel, and its material properties are set to only write to the depth buffer and not output color values.

[0104] S202 executes the static wide-angle channel rendering process.

[0105] The dual-view frustum blending rendering module 200 first starts the static rendering thread. This thread reads the preset fixed viewpoint coordinates E. fix This coordinate is typically set at the center of the area in front of the immersive display terminal, covering the densest area of ​​the crowd. Based on E fix Based on the physical dimensions of the immersive display terminal, a standard perspective projection matrix is ​​constructed. The field of view of the matrix is ​​set to cover the entire display screen area.

[0106] use For the environmental background object G bg Rasterization rendering is performed, and the generated color data is stored in the static frame buffer (FB). stat In the middle. The image generated at this time is for the location E fix Nearby observers have the correct spatial perspective.

[0107] S203, calculate dynamic off-axis projection parameters.

[0108] The dual-view frustum blending rendering module 200 starts the dynamic rendering thread. This thread reads the real-time observer viewpoint coordinates P output by the virtual-real space calibration module 100. user To ensure that the observer interacts with the foreground object G during movement... dyn To consistently present the correct 3D floating effect, the system uses an off-axis projection algorithm to dynamically construct an asymmetric view frustum.

[0109] The specific calculation process is as follows:

[0110] First, obtain the basis vectors of the screen plane in the virtual space. Let V be the coordinates of the bottom left corner of the screen. bl The coordinates of the lower right corner are V br The coordinates of the upper left corner are V tl .

[0111] Screen horizontal basis vectors The calculation is as follows:

[0112]

[0113] Screen vertical basis vectors The calculation is as follows:

[0114]

[0115] Screen normal vector The calculation is as follows:

[0116]

[0117] Next, the observer's viewpoint P user Project onto the screen space defined by the basis vectors and calculate the view frustum boundary parameters.

[0118] Define the relative vector from the viewpoint to the bottom left corner of the screen.

[0119]

[0120] Calculate the vertical distance n from the viewpoint to the screen plane. dist This distance serves as the reference distance for the near clipping plane in the projection calculation:

[0121]

[0122] Based on reference distance n dist Given a set distance n from the near clipping plane, calculate the four boundary values ​​of the projection plane: left boundary l, right boundary r, lower boundary b, and upper boundary t.

[0123]

[0124] S204, construct and render the dynamic off-axis projection matrix.

[0125] Based on the boundary parameters obtained in step S203, the off-axis projection matrix of the dynamic channel is constructed.

[0126] Where f is the distance to the far cutting surface.

[0127] use For interactive foreground object G dynand phantom geometry G phan Rendering is then performed. During this process, the phantom geometry G... phan Plays a key role: Although invisible, it participates in depth testing and shadow calculation. When a virtual light source illuminates G... dyn At that time, the shadow casting algorithm will calculate G dyn In G phan The shadowed area produced on the surface.

[0128] The rendering result is output to the dynamic frame buffer (FB). dyn FB dyn It contains two parts of data: the color information of the foreground object and the shadow masking information. The shadow masking information records the correct shadow shape of the dynamic object projected onto the background geometry, and this shadow shape is based on the real-time viewpoint P. user The calculations ensure that the spatial contact logic between dynamic objects and static backgrounds is continuous in terms of visual perception.

[0129] Please see the appendix Figure 1 -Appendix Figure 2 The bidirectional inverse projection lighting processing module 300 is configured to establish a light and shadow mapping relationship between the physical lighting environment and the virtual heterogeneous rendering channel. This process specifically includes the following sub-steps:

[0130] S301, acquire and parse the physical lighting vector.

[0131] The bidirectional inverse projection lighting processing module 300 receives the physical light source reference direction vector, which has been transformed into the virtual scene coordinate system, transmitted from the virtual-real space calibration module 100. This vector represents the incident direction of the main light source in the physics exhibition hall relative to the origin of the virtual world.

[0132] S302, calculate the first illumination transformation matrix of the dynamic channel.

[0133] For dynamic off-axis rendering channels, since the camera position follows the observer's viewpoint P in real time... user Its observation space is consistent with the head movement of the physical observer. Therefore, in order to match the highlight and shadow directions of the surface of the dynamic interactive object with the lighting of the physical environment, the bidirectional inverse projection lighting processing module 300 constructs a first lighting transformation matrix.

[0134] In this embodiment, the first illumination transformation matrix This can be an identity matrix or a lighting enhancement matrix based on specific scene requirements. The system directly uses the reference direction vector. Applied to shaders with dynamic channels:

[0135]

[0136] in, The final lighting vector used for the dynamic channel. This means that when the physical light source is located in the upper left of the screen, the highlight of the interactive object will always appear in the upper left area of ​​its geometry, which is physically intuitive, no matter where the observer moves.

[0137] S303 calculates the parallax angle and inverse view compensation matrix.

[0138] For the static wide-angle rendering pass, the camera position is locked at a fixed viewpoint E. fix When the real-time observer is located at P user When using it directly During rendering, because the perspective of the background is fixed, the observer will perceive a discrepancy between the light and shadow directions of the background and those of the foreground objects. To eliminate this visual light and shadow discontinuity, the bidirectional inverse projection lighting processing module 300 calculates the inverse perspective compensation matrix M. comp .

[0139] First, calculate the scene center O center Two line-of-sight vectors refer to the reference point. Static line-of-sight vectors. Defined as:

[0140]

[0141] Dynamic gaze vector Defined as:

[0142]

[0143] Next, calculate from Rotate to Rotation axis and rotation angle θ:

[0144]

[0145] Constructing an inverse perspective compensation matrix based on Rodrigues' rotation formula. This matrix describes the spatial orientation differences caused by changes in viewpoint.

[0146] S304 generates and applies the second illumination transformation matrix for the static channel.

[0147] The bidirectional inverse projection lighting processing module 300 utilizes the inverse viewpoint compensation matrix M comp For the reference direction vector Perform an inverse rotation transformation to obtain the corrected illumination vector used in the static channel.

[0148] This corrected lighting vector This light is passed to the shader in the static wide-angle rendering pass. The technical effect of this step is that the system artificially distorts the lighting direction of the background environment. When the observer stands off-center at point P... user When viewing the screen from a certain position, although the geometric perspective of the background is incorrect, the direction of light and shadow, after inverse correction, will tend to align with the direction of light and shadow of the foreground objects in the observer's retinal image. This parallax-based lighting compensation mechanism effectively bridges the spatial discontinuity caused by dual-view frustum rendering at the visual psychological level.

[0149] Please see the appendix Figure 1 The interactive synthesis module 400 performs cross-channel spatial interactive calculations and image fusion processing. The specific steps are as follows:

[0150] S401 performs spatial collision detection.

[0151] The interactive synthesis module 400 receives the coordinates P of the virtual interactive node. inter The system retrieves the interactive foreground object G from the dynamic channel. dyn The system first obtains the geometric mesh data of P. inter Convert to G dyn Under the same local coordinate system of the model, or with G dyn The vertex data is transformed to the world coordinate system to ensure consistency in distance calculation benchmarks. The module constructs the interactive foreground object G. dyn The axis is aligned with the bounding box. When P inter After entering the bounding box, traverse the mesh vertices and calculate the Euclidean distance d between the interactive node and each vertex. i If there exists a vertex v i Make d i Less than the preset collision threshold δ coll If so, it is determined that an interactive contact has occurred.

[0152] S402, calculates mesh deformation based on Gaussian decay.

[0153] During the vertex shader phase, the interactive foreground object G... dyn Perform mesh vertex displacement. The displacement is calculated using Gaussian radial basis functions to simulate surface depressions or bulges under stress. Vertex displacement vector. The calculation formula is:

[0154]

[0155] Among them, P i The original coordinates of the vertex; A is the original normal vector of the vertex; ampσ is the deformation amplitude coefficient, whose sign determines the deformation direction, and its absolute value is positively correlated with the interaction speed; σ is the attenuation constant of the deformation influence radius. Updated vertex position P′ i for:

[0156]

[0157] This step involves modifying geometric data to induce physical deformation in dynamic objects.

[0158] S403, Perform cross-channel normal redirection.

[0159] Interactive synthesis module 400 calculates the tangent space normal perturbation caused by deformation. This perturbation is derived based on the spatial gradient of the Gaussian deformation function described above. Specifically, the partial derivative of the displacement function with respect to the tangent direction is calculated to construct the perturbation vector:

[0160]

[0161] The system normalizes the calculated perturbation vector, maps it to RGB color values, and writes it to a shared normal perturbation texture.

[0162] Draw the environment background object G in the static wide-angle rendering channel. bg At that time, the fragment shader samples the shared normal perturbation texture, obtains the perturbation amount at the corresponding texture coordinates, and superimposes it onto the original normal of the background. Above, calculate the final coloring normal.

[0163]

[0164] Where λ is the perturbation intensity blending factor. Through this step, the deformation data of the foreground object is converted into normal data and passed to the background rendering pipeline, so that the background produces lighting changes that match the deformation of the foreground.

[0165] S404, Image synthesis based on shadow masking.

[0166] Interactive compositing module 400 acquires static frame buffer FB stat Color data C stat Dynamic Frame Buffer (FB) dyn Color data C dyn And shadow masking data M mask Shadow mask data M mask This is single-channel data, and its value represents the degree to which light is blocked. The final composite image color C final The calculation formula is as follows:

[0167] C final =Cstat ·(1-α shadow ·(1-M mask ))+C dyn ;

[0168] Where α shadow This is the shadow density coefficient. The formula first uses shadow masking to reduce the brightness of the corresponding pixels of the static background, simulating the shadow cast by the foreground on the background, and then overlays the color data of the dynamic foreground object onto the processed background.

[0169] S405, screen geometric distortion correction.

[0170] The interactive compositing module 400 performs two-dimensional geometric correction on the non-planar physical structure of the immersive display terminal. The system calls a preset screen geometry lookup table, which records the pixel coordinates (u...) of the output image. out ,v out ) and input synthetic image texture coordinates (u in ,v in Mapping relationship between )

[0171] (u in ,v in = LUT(u out ,v out );

[0172] The interactive synthesis module 400 performs C according to this mapping relationship. final Resampling is performed to generate a corrected image that can offset the visual distortion caused by the physical curvature of the screen, and then sent to the immersive display terminal via the video interface.

[0173] Please see the appendix Figure 3 This invention provides a method for interactive multimedia content in exhibition halls based on naked-eye 3D, comprising the following steps:

[0174] S1, The mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system is established through the virtual and real space calibration module 100, and the observer coordinate data and physical light source data collected by the multi-dimensional perception sensor group are converted into parameters under the virtual scene coordinate system.

[0175] S2, through the dual-view frustum hybrid rendering module 200, constructs a static wide-angle rendering channel and a dynamic off-axis rendering channel, and loads the corresponding scene geometry model in the two channels respectively;

[0176] S3, the bidirectional inverse projection lighting processing module 300 calculates the first lighting transformation matrix acting on the static wide-angle rendering channel and the second lighting transformation matrix acting on the dynamic off-axis rendering channel based on physical light source data and viewpoint parallax angle.

[0177] S4, parallel rendering is performed through the dual-view frustum hybrid rendering module 200. The static wide-angle rendering channel outputs the background image based on a fixed viewpoint, and the dynamic off-axis rendering channel outputs the foreground image and shadow mask based on the real-time observer viewpoint and the second lighting transformation matrix.

[0178] S5 detects the collision state between the hand interaction coordinates and the virtual object through the interactive compositing module 400, calculates the mesh deformation data and transmits it to the rendering channel for normal update;

[0179] S6, through the interactive compositing module 400, combines the background image, foreground image, and shadow mask to perform image compositing and geometric correction, generating the final display screen.

[0180] The method in this embodiment can be used to execute the above system embodiment, and its principle and technical effect are similar, so it will not be described again here.

Claims

1. An immersive multimedia content interactive system for exhibition halls based on naked-eye 3D, characterized in that, include: The virtual-real space calibration module is configured to establish a mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system, and to convert the observer's viewpoint coordinates and physical light source vectors collected by the multi-dimensional perception sensor group into parameters under the virtual rendering scene coordinate system. The dual-view frustum hybrid rendering module is configured to construct a static wide-angle rendering channel and a dynamic off-axis rendering channel in parallel; wherein, the static wide-angle rendering channel renders the environment background object based on a preset fixed viewing point, and the dynamic off-axis rendering channel renders the interactive foreground object based on the viewer's viewpoint coordinates; The bidirectional inverse projection lighting processing module is configured to calculate, based on the physical light source vector, the first lighting transformation parameters acting on the static wide-angle rendering channel and the second lighting transformation parameters acting on the dynamic off-axis rendering channel, respectively. The interactive compositing module is configured to fuse the background image generated by the static wide-angle rendering channel with the foreground image generated by the dynamic off-axis rendering channel based on the observer's viewpoint coordinates and the collision state of the interactive foreground object, and then output the fusion to the immersive display terminal.

2. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D as described in claim 1, characterized in that, When constructing the dynamic off-axis rendering channel, the dual-view frustum hybrid rendering module performs off-axis projection calculations: Obtain the corner coordinates of the screen plane of the immersive display terminal in the virtual rendering scene coordinate system, and construct the basis vector of the screen plane; Project the observer's viewpoint coordinates onto the screen space defined by the basis vectors, and calculate the vertical distance and relative offset of the observer's viewpoint coordinates relative to the screen plane; Based on the vertical distance and relative offset, the left and right boundary values ​​and the top and bottom boundary values ​​of the asymmetric view frustum are determined, and then an off-axis projection matrix is ​​generated for rendering the interactive foreground object.

3. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D as described in claim 1, characterized in that, The dual-view frustum hybrid rendering module also loads phantom geometry in the dynamic off-axis rendering channel; The phantom geometry is a geometric copy of the environment background object. The phantom geometry is configured as an invisible material and only participates in depth testing and shadow calculation. The dynamic off-axis rendering channel uses the phantom geometry to receive the shadows cast by the interactive foreground object and generates a shadow mask containing only shadow information. The interactive compositing module uses the shadow mask to modulate the brightness of the background image generated by the static wide-angle rendering channel.

4. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D as described in claim 1, characterized in that, The bidirectional inverse projection lighting processing module calculates the second lighting transformation parameters in the following way: Obtain the physical light source reference direction vector after conversion by the virtual-real space calibration module; The physical light source reference direction vector is directly mapped to the light source incident direction in the dynamic off-axis rendering channel, so that the lighting direction of the interactive foreground object is consistent with the actual lighting direction of the physical exhibition hall.

5. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D according to claim 4, characterized in that, The bidirectional inverse projection lighting processing module calculates the first lighting transformation parameters in the following way: Calculate the parallax rotation angle between the dynamic line-of-sight vector corresponding to the observer's viewpoint coordinates and the static line-of-sight vector corresponding to the fixed observation viewpoint, with the scene center as the reference point. Construct an inverse viewpoint compensation matrix based on the parallax rotation angle; The physical light source reference direction vector is rotated inversely using the inverse perspective compensation matrix to generate a corrected lighting vector, which is then used as the light source incident direction of the static wide-angle rendering channel.

6. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D as described in claim 1, characterized in that, The interactive synthesis module is also configured to perform mesh deformation calculations: When the interactive node corresponding to the observer's viewpoint coordinates is detected to enter the collision bounding box of the interactive foreground object, the distance between the interactive node and the vertex of the interactive foreground object surface is calculated. The vertex displacement is calculated using the Gaussian radial basis function, and the vertex displacement decreases non-linearly with the increase of the distance. The vertex displacement is applied during the vertex shader stage to modify the mesh geometry of the interactive foreground object in real time.

7. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D according to claim 6, characterized in that, The interactive composition module is also configured to perform cross-channel normal redirection: Calculate the tangent space normal perturbation vector of the interactive foreground object caused by mesh deformation; Write the normal perturbation vector into the shared normal perturbation texture; When rendering the environment background object, the static wide-angle rendering channel samples the normal perturbation texture and superimposes the sampled perturbation vector onto the original normal of the environment background object, thereby changing the surface lighting and reflection properties of the environment background object.

8. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D according to claim 3, characterized in that, The specific method by which the interactive synthesis module fuses images is as follows: Obtain the color value of the background image, the color value of the foreground image, and the grayscale value of the shadow mask; The grayscale values ​​of the shadow mask are used to perform a weighted multiplication of the color values ​​of the background image to obtain a darkened background image; The color values ​​of the foreground image are superimposed onto the darkened background image to obtain a composite image; The screen geometry correction lookup table is invoked to perform two-dimensional nonlinear resampling on the synthesized image, generating a final output frame that adapts to the physical curvature of the immersive display terminal.

9. The immersive multimedia content interaction system for exhibition halls based on naked-eye 3D according to claim 1, characterized in that, The virtual-real space calibration module stores a rigid body transformation calibration matrix; The virtual-real space calibration module transforms the physical light source vector through the rotation component of the rigid body transformation calibration matrix, and transforms the observer's viewpoint coordinates through the rotation and translation components of the rigid body transformation calibration matrix.

10. A method for immersive multimedia content interaction in exhibition halls based on naked-eye 3D, comprising the steps described in any one of claims 1-9: S1. Establish the mapping relationship between the physical exhibition hall coordinate system and the virtual rendering scene coordinate system through the virtual and real space calibration module, and convert the collected observer viewpoint coordinates and physical light source vectors into parameters under the virtual rendering scene coordinate system. S2. Construct a static wide-angle rendering channel and a dynamic off-axis rendering channel in parallel using a dual-view frustum hybrid rendering module; S3. Based on the physical light source vector, the bidirectional reverse projection lighting processing module calculates the first lighting transformation parameters acting on the static wide-angle rendering channel and the second lighting transformation parameters acting on the dynamic off-axis rendering channel, respectively. S4. Control the static wide-angle rendering channel to render the background image based on a fixed viewing point and the first lighting transformation parameters, and control the dynamic off-axis rendering channel to render the foreground image based on the observer's viewpoint and the second lighting transformation parameters. S5. The background image and the foreground image are merged through the interactive synthesis module and output to the immersive display terminal.

Images