A virtual reality-based immersive digital media art display method and system

Abstract

This invention relates to the intersection of virtual reality technology and digital media art display, and discloses an immersive digital media art display method and system based on virtual reality. The system comprises a VR headset, dual handheld controllers, a computer host, and a software system running on the host. By constructing display elements such as virtual exhibition halls, color palettes, and canvases, it achieves seamless switching between browsing and creation modes using the controller's grip buttons, establishes a precise mapping from the controller's six degrees of freedom pose to the virtual brush, realizes color picking and 3D drawing, synchronously generates spatialized audio that changes with operation, and supports brush-level data saving and playback of the creation process. This invention empowers users with autonomous creation capabilities, eliminates the separation of visual and auditory senses, establishes a multi-sensory synchronous feedback loop, fully records and reproduces the creation process, and significantly enhances the immersion and interactivity of digital media art display.

Description

Technical Field

[0001] This invention relates to the intersection of virtual reality technology and digital media art display, specifically to an immersive digital media art display method and system based on virtual reality. Background Technology

[0002] With the improvement of VR hardware performance and the advancement of cultural digitization, virtual reality-based digital media art exhibitions have become an important form of digital cultural dissemination. Currently, several technical solutions have been developed for this field, exploring personalized displays. These solutions can dynamically adjust content presentation based on user behavior preferences, adjusting content based on changes in user emotions by constructing emotional state curves and interest concentration curves. They also support gesture control and haptic feedback, enabling user interaction with virtual patterns.

[0003] However, the existing solutions mentioned above still have many shortcomings: First, there is a lack of independent creation tools. The existing system functions are all centered around browsing and recommending preset art content, without providing a mechanism to transform the handheld controllers into paintbrushes for real-time creation in three-dimensional space. This keeps users in the role of passive viewers, unable to become creators. Second, there is a disconnect between operation and sound effects. The audio in the system consists of preset background music or fixed voice. When users perform operations such as picking colors and drawing brush strokes, there is no corresponding spatial sound feedback, resulting in a significant disconnect between visual and auditory information and greatly reducing the immersive experience of the presentation. Finally, the mapping is inaccurate and lacks multi-sensory synchronization. The existing solution does not establish a precise mathematical mapping from the six degrees of freedom pose of the handheld controllers to the position and orientation of the virtual paintbrush tip, nor does it simultaneously create a closed loop of synchronous feedback between user operations and spatial sound effects and tactile vibrations, affecting the user's operating experience and interaction accuracy.

[0004] Therefore, in view of the above situation, there is an urgent need to develop an immersive digital media art display method and system based on virtual reality to overcome the shortcomings in current practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for immersive digital media art display based on virtual reality, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An immersive digital media art display method based on virtual reality includes the following steps: S1: Construct a virtual exhibition environment that includes a virtual showroom, color palette, and canvas; S2: Real-time acquisition of six-DOF pose information and button status information of the VR headset worn by the user and the controllers held in both hands; S3: Seamlessly switch between browsing mode and creation mode based on the grip button status of the right hand holding the controller; S4: In creation mode, the pose information of the right hand holding the controller is mapped to the pose information of the virtual brush. S5: Responds to the trigger button operation of the right hand holding the controller, picks up colors on the color palette and draws them in three-dimensional space; S6: Based on color picking, drawing, and brush strokes, complete the operation events, generate corresponding spatialized audio, and provide feedback to the user; S7: Save the brushstroke-level data during the drawing process and support playback of the creation process based on the brushstroke-level data.

[0007] As a further aspect of the present invention: in step S3, when it is detected that the grip button of the right hand-held controller is continuously pressed for more than a preset time, the system switches from browsing mode to creation mode. When the system detects that the grip button has been released, it automatically reverts to browsing mode.

[0008] As a further aspect of the present invention: in step S4, the position of the virtual pen tip is obtained by extending a preset distance in front of the position of the right hand holding the controller, and the orientation of the virtual pen is consistent with the orientation of the right hand holding the controller.

[0009] As a further aspect of the present invention: in step S6, the generation of spatialized audio includes: The relative orientation is calculated based on the location of the sound source of the operation event and the user's head position, and the pre-stored audio samples are spatially rendered based on the head-related transfer function. The audio playback rate corresponding to the drawing operation is positively correlated with the pen stroke speed of the virtual brush.

[0010] As a further aspect of the present invention: in step S7, the brushstroke-level data is saved in a point sequence format containing three-dimensional coordinates and color information; When the creation process is replayed, the process of generating each brushstroke is reproduced stroke by stroke in the original time sequence, and users can freely adjust the viewing angle during the replay.

[0011] An immersive digital media art display system based on virtual reality, comprising: VR headsets are used to present immersive virtual scenes to users; A dual-hand controller is used to collect the user's hand operation commands and six-degree-of-freedom pose information; A computer host is communicatively connected to both the VR headset and the dual handheld controllers. The computer host runs a software system, which includes: The Display Elements Building Module is used to construct a virtual display environment that includes a virtual exhibition hall, color palette, and canvas. The data acquisition module is used to collect the posture information and button status information of the VR headset and the two handheld controllers in real time. The mode switching module is used to seamlessly switch between browsing mode and creation mode based on the gripping state of the right hand holding the controller. The brush mapping module is used to map the pose information of the right hand holding the controller to the pose information of the virtual brush in creation mode. The interactive drawing module is used to respond to the trigger button operation of the right hand holding the controller, pick up colors on the color palette and draw in three-dimensional space; The audio generation module is used to generate corresponding spatial audio based on color picking, drawing, and brush strokes to complete operation events and provide feedback to the user through the VR headset. The save and playback module is used to save the brushstroke-level data during the drawing process and supports the playback of the creation process based on the brushstroke-level data.

[0012] As a further aspect of the present invention: the mode switching module is specifically used for: When the right hand holding the controller's grip button is detected to be pressed continuously for more than a preset time, the control system switches from browsing mode to creation mode; When the grip button is detected to be released, the control system automatically returns to browsing mode.

[0013] As a further aspect of the present invention: the brush mapping module is specifically used for: Calculate the forward direction vector of the right-hand controller, and extend the position of the right-hand controller along the forward direction vector by a preset distance to obtain the position of the virtual pen tip. The orientation of the virtual pen is consistent with the orientation of the right-hand controller.

[0014] As a further aspect of the present invention: the audio generation module is specifically used for: The relative orientation is calculated based on the location of the sound source of the operation event and the user's head position, and the pre-stored audio samples are spatially rendered based on the head-related transfer function. The audio playback rate corresponding to the drawing operation is positively correlated with the pen stroke speed of the virtual brush.

[0015] As a further aspect of the present invention: the save and playback module is specifically used for: Save the stroke-level data in a point sequence format that includes three-dimensional coordinates and color information; When the creation process is replayed, the process of generating each brushstroke is reproduced stroke by stroke in the original time sequence, and users can freely adjust the viewing angle during the replay.

[0016] Compared with the prior art, the beneficial effects of the present invention are: It empowers users with the ability to create independently. By pressing and holding the right handle grip button for 0.5 seconds, users can seamlessly switch between browsing mode and creation mode. The handle pose is accurately converted into pen tip coordinates through a 0.15m extension vector. Combined with the trigger button, color picking and 3D drawing are realized. Users can create freely in 3D space at any time, which fundamentally solves the functional defects of existing systems that lack creation tools. It effectively eliminates the problem of visual and auditory disconnect. When users pick colors, draw, and complete brush stroke operations, corresponding sound effects will be triggered. The sound source is spatialized by HRTF, and the drawing speed can also affect the sound effect playback speed in real time, so that the hearing and operation behavior are strictly synchronized in time and space. A multi-sensory precision mapping mechanism is established, which realizes accurate mapping from handle pose to pen tip coordinates through precise mathematical calculations. Combined with the real-time generation of 16mm strip-shaped surface, 50ms vibration for color pickup confirmation, and HRTF sound effects for real-time tracking of pen stroke trajectory, the three senses of sight, touch, and hearing form a synchronous feedback loop, making the operation feel close to real painting. It enables the complete preservation and playback of the creative process. Brushstrokes are saved in JSON format as point sequences containing three-dimensional coordinates and colors, which is superior to the existing method of only saving the final image. During playback, the creative process is reproduced stroke by stroke in the original time sequence, providing a solid foundation for the dissemination and teaching of digital works. Attached Figure Description

[0017] Figure 1 This is an architecture diagram of an immersive digital media art display system based on virtual reality, as described in an embodiment of the present invention.

[0018] Figure 2 This is a flowchart of an immersive digital media art display method based on virtual reality in an embodiment of the present invention.

[0019] In the diagram: solid arrows represent the real-time processes automatically executed by the system, the mandatory timing transitions, and the necessary execution steps of 60Hz cyclic acquisition; The dashed arrows represent optional execution steps that are triggered only when the user actively issues a save command; if no command is issued, the step will not be executed. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0022] Please see Figure 1 and Figure 2 This invention provides a method and system for immersive digital media art display based on virtual reality. The system consists of a VR headset, dual handheld controllers, a computer host, and a software system running on the host. The VR headset is either a PICO4 or Meta Quest3 with a refresh rate ≥90Hz, used to present an immersive virtual scene to the user. The dual handheld controllers have a built-in IMU, support 6DoF tracking, and have trigger and grip buttons for collecting user hand gestures and pose information. The computer host has a CPU ≥3.0GHz, memory ≥16GB, and a graphics card ≥RTX4070, used to run the software system, process pose data, and generate rendered images and spatialized audio.

[0023] The VR headset, dual handheld controllers, and computer host are all conventional devices in the field, and their connection methods and data transmission protocols adopt standard VR device connection methods known in the field, such as wireless / wired data transmission via USB-C or Wi-Fi 6E.

[0024] In one embodiment of the present invention, the virtual exhibition hall and display elements are constructed as follows: A 15m×10m×3.5m cube-shaped exhibition hall was constructed in the VR space, with each of the four walls measuring 12m×2.8m. Thirty-six digital artworks, each measuring 2.5m×1.8m, were loaded onto the walls using a 3×3 grid. This layout of the exhibition hall and walls provides users with ample browsing space while ensuring the clarity and visual impact of the digital artworks.

[0025] A 1.5m x 1.0m color palette is placed 1.2m above the ground in the center of the exhibition hall. It contains 128 square color blocks of 40mm x 40mm arranged in 8 rows x 16 columns. The colors are encoded in HSV: the rows vary the hue (uniformly distributed from 0° to 360°), and the columns vary the saturation and brightness. This color palette layout can cover the color range required by mainstream digital painting, and the size and spacing of the color blocks are adapted to the hand operation precision in VR environment, making it easy for users to quickly pick up the target color.

[0026] A 3.0m x 2.0m white rectangular canvas is set up in one corner of the exhibition hall, with vertex world coordinates of (0,0,0), (3,0,0), (3,2,0), and (0,2,0), and the normal is in the positive Z-axis direction. "Browse Mode" and "Creative Mode" status indicators are displayed 0.3m above the left handle, allowing users to easily understand the current system operating mode.

[0027] The system acquires 6DoF pose (position coordinates and normalized quaternions) of the head and dual controllers, as well as the Boolean states of the trigger and grip buttons, via an OpenXR interface at a frequency of 60Hz. The world coordinate system is based on the user's standing position at startup, with the X-axis to the right, the Y-axis upward, and the Z-axis forward. The OpenXR interface is a universally accepted standard interface for VR device interaction in this field, enabling unified data acquisition and control across different brands of VR devices. The high-frequency pose acquisition at 60Hz ensures smoothness in subsequent virtual brush mapping and drawing processes, avoiding screen stuttering and operational delays.

[0028] In one embodiment of the present invention, the creation mode is mapped to a virtual paintbrush: 1. Mode Switching The system monitors the status of the right handle's grip button in real time. When the grip button is detected to be pressed for more than 0.5 seconds, the system switches from browsing mode to creation mode; when the grip button is detected to be released, the system automatically returns to browsing mode. This simple button operation achieves seamless mode switching without requiring complex menu operations, ensuring the continuity of the user's creative flow and solving the problem of existing systems being unable to quickly switch between browsing and creation modes.

[0029] 2. Pen pose calculation In creation mode, the system obtains the position of the right controller. And the orientation quaternion, after converting the orientation quaternion into a rotation matrix, take the Z-axis direction vector. The pen tip position is calculated by extending 0.15m forward from the handle position, using the following formula:

[0030] in, The three-dimensional coordinates of the virtual brush tip along the X / Y / Z axes in the world coordinate system, in meters (m). : The three-dimensional position coordinates of the right-hand hand controller on the X / Y / Z axes in the world coordinate system, in meters; The X / Y / Z components of the Z-axis direction vector after the controller is held in the right hand and the quaternion is converted into a rotation matrix.

[0031] The brush is oriented in the same direction as the handle's Z-axis. In VR, it is rendered as a cylinder with a length of 0.15m and a diameter of 0.005m, and its color is the current brush color. Through the precise mathematical mapping relationship described above, the user's real hand movements can be accurately converted into the pose of the virtual brush, solving the problem of inaccurate mapping between the handle pose and the virtual brush in existing technologies, and making virtual painting operations closer to the feel of real painting.

[0032] 3. Color Picking When the vertical projection distance from the brush tip to the palette panel is less than 0.05m, and the projection point falls within a 40mm x 40mm area of ​​a certain color block, if the user presses the right handle trigger button, the system will set the HSV value of that color block to the current brush color. Simultaneously, the system triggers vibration on the right handheld controller (50ms, maximum intensity 30%) and sends a "pick up color" event to the audio module, the event including the pickup location. and color RGB values.

[0033] The aforementioned color picking mechanism enables precise color selection. Combined with tactile vibration feedback, it allows users to intuitively perceive the completion status of the color picking operation, enhancing the certainty of the operation and the interactive experience.

[0034] 4. 3D rendering In creative mode, if the trigger button is held down and the pen tip is more than 0.1m from the color palette panel, the system enters drawing mode. The system records the pen tip position at 60Hz. This forms a sequence of dots, each dot carrying the current brush color. A strip-shaped patch is generated in real time between each adjacent sampling point. The specific calculation process is as follows: Calculate the direction vectors of adjacent sampling points: ; Where d: the pen stroke direction vector between two adjacent sampling points; : The three-dimensional position coordinates of the virtual brush tip at the (i+1)th sampling time; : The three-dimensional position coordinates of the virtual brush tip at the i-th sampling time.

[0035] Calculate the offset vector: If d is parallel to the Z-axis, then... ; in, : An offset vector perpendicular to the pen stroke direction (used to generate the width of the strip patch); normalize(): Vector normalization function.

[0036] The strip-shaped patch has a width of 0.016m, and the coordinates of its four vertices are as follows: in, : The three-dimensional coordinates of the four vertices of the strip surface.

[0037] The color of the strip area is the current brush color. With an opacity of 0.9, it is overlaid onto the canvas in a blending mode of "source × 0.9 + destination × 0.1". For each surface generated, the system sends a "draw" event to the audio module, carrying the current pen tip position and color.

[0038] When the user releases the trigger button, the system exits the drawing state, calculates the total length of the stroke (the sum of the Euclidean distances of all adjacent sampling points) and the average color (averaging the RGB three channels respectively), and sends a "stroke complete" event to the audio module.

[0039] The real-time strip patch generation algorithm described above can achieve smooth 3D rendering effects with fixed brush stroke width and smooth edges. The overlay mode can simulate the mixing effect of real pigments, making virtual paintings more layered and realistic.

[0040] 5. Spatialized audio generation The system preloads three 44.1kHz / 16bit mono audio samples: pickup.wav (0.2 seconds of "plop" sound), drawing_base.wav (1.0 second of friction sound, variable speed loop), and complete.wav (0.5 seconds of "ding" sound).

[0041] The audio processing flow is as follows: The system determines the location of the sound source in the event. With the user's head position Calculate the relative azimuth vector Convert the relative azimuth vector into azimuth angle θ and elevation angle. The left and right ear filter coefficients are obtained by querying the MIT KEMAR HRTF database (5° resolution). The mono audio is then convolved with the filters to output spatialized stereo sound. The MIT KEMAR HRTF database is a recognized high-precision head-related transfer function database in the field, capable of accurately simulating the acoustic characteristics of sound reaching the human ear from different directions.

[0042] Pick up sound effects: When a "pick up color" event is received, the system picks up the sound effect at the location. Play pickup.wav once for the audio source.

[0043] Drawing sound effects: When a "drawing" event is received, the system loops the drawing_base.wav file, using the current pen tip as the sound source. The playback rate is determined by the pen stroke speed, calculated using the following formula: ; Where, vmove: the real-time brush speed of the virtual brush, in m / s; 0.0167s: the sampling time interval corresponding to the system's 60Hz sampling frequency, in s; ; Where, rate: real-time playback rate of spatial audio, unit: speed multiplier; The playback rate ranges from 0.5 to 2.5 times. The system uses 20ms crossfade-in and fade-out to eliminate loop noise and ensure the continuity of sound effects.

[0044] Complete sound effect: When the "Complete stroke" event is received, the system stops drawing sound effects and plays complete.wav once with the last stroke position as the sound source.

[0045] Through the aforementioned spatial audio generation mechanism, strict synchronization between operation and sound effects in time and space can be achieved. The correlation between the rendering rate and the sound effect playback rate can further enhance the realism of the operation, eliminate the problem of the separation of visual and auditory information in the existing technology, and significantly improve the immersive experience.

[0046] In one embodiment of the present invention, rendering and work saving and playback are performed as follows: 1. Real-time rendering The system employs stereoscopic rendering technology, with the interpupillary distance of the left and right virtual cameras set to 0.063m and a rendering frame rate ≥72FPS. Strip-shaped patches are drawn sequentially with a blending transparency of 0.9 to ensure the accuracy of brushstroke overlay effects. These rendering parameter settings provide users with a clear and smooth stereoscopic visual effect, avoiding dizziness while ensuring real-time rendering.

[0047] 2. Preservation of works When the user presses the left trigger button, the system collects the point sequence data of all brush strokes on the canvas; each brush stroke contains... The system serializes the above data into a JSON file and writes it to the %APPDATA% / VRArtGallery / saves / directory, along with an array of points (each element being [x,y,z,R,G,B]) and a timestamp. Simultaneously, the system renders a snapshot of the canvas at a resolution of 2048×1536 and saves it as a PNG file.

[0048] Unlike existing technologies that only save the final image, this invention can completely record the user's creative process by saving brushstroke point sequence data containing three-dimensional coordinates and colors, providing a foundation for the dissemination, modification, and teaching of digital works.

[0049] 3. Playback of the work In browsing mode, the system adds thumbnails of user works to the exhibition wall. When a user is within 2 meters of the exhibition wall, they can click the thumbnail using the right handgrip trigger. The system loads the corresponding JSON file and displays a 1.2m x 0.9m floating panel 1.5m directly in front of the user, showing a snapshot of the work. The strip-shaped panels are then replayed sequentially at 0.03-second intervals according to timestamps, providing an animated playback of the creative process. Users can freely move their viewpoint during playback to observe the creative process from different angles.

[0050] The aforementioned playback function allows other users to intuitively understand the creative ideas and drawing process of the work, enriching the display forms and dissemination dimensions of digital media art. Specific Implementation

[0051] The user wears the PICO4 headset, holds 6DoF controllers in both hands, and connects to a computer configured with an i7-13700K / 32GB / RTX4070 via USB-C. After the system boots up, a world coordinate system is established with the user's standing position as the origin.

[0052] Browsing Phase: Users freely walk through a 15m x 10m virtual exhibition hall, viewing 36 digital artworks on the four walls. When a user is drawn to a warm-toned abstract painting, they develop a creative urge and head towards the color palette in the center of the hall.

[0053] Creative Mode Switching: Press and hold the right handle grip button for 0.5 seconds. A white virtual brush (0.15m long cylinder) will appear in front of the right hand, and the "Creative Mode" icon on the top of the left handle will be highlighted. The system has successfully switched to creative mode.

[0054] Color Picking: The user moves their right hand to align the pen tip with the warm orange color block on the color palette panel. At this point, the pen tip is approximately 0.03m from the panel, and the projection point falls within the area of ​​that color block. The user pulls the trigger button with their index finger, and the virtual brush color changes to warm orange. Simultaneously, the right handle vibrates briefly for 50ms, and a clear "pop" sound is heard in the headphones from the direction of the color block, indicating that the color picking operation is complete.

[0055] 3D rendering: The user keeps the right handle grip button pressed, turns to face the white canvas in a corner of the exhibition hall, pulls the trigger button again and slowly moves their right hand. The system enters rendering mode, recording the pen tip trajectory at a frequency of 60Hz. The pen tip moves from (0.85, 0.92, 0.02) through continuous sampling points to (2.75, 1.95, 0.02). Between every two adjacent sampling points, the system generates a 16mm wide warm orange semi-transparent strip in real time, and overlays it onto the canvas in a blending mode of "source × 0.9 + target × 0.1". The brushstrokes smoothly "grow" as the pen tip moves.

[0056] Simultaneously, the sound of chalk scratching is heard through the headphones, the sound source closely following the movement of the pencil tip. As the user's arm moves faster, the pen speed increases from 0.5m / s to 0.9m / s, the sound effect playback rate increases from 1.0x to approximately 1.8x, and the pitch rises accordingly; when the user slows down, the sound effect playback rate decreases, and the pitch becomes lower. When the user releases the trigger, the scratching sound immediately stops, and a "ding" sound is heard from the end of the stroke, indicating that the drawing is complete in one stroke.

[0057] Artwork Saving and Playback: The user repeats the color picking and drawing steps described above, picking up a total of 8 colors and drawing 20 brushstrokes to complete an abstract digital painting. The user presses the left handle trigger button, and the system saves the 3D coordinates, color data, and timestamps of all brushstrokes as a JSON file, generating a 2048×1536 resolution PNG snapshot. The user releases the right handle grip button, and the system returns to browsing mode, adding a thumbnail of the artwork to the display wall. The user clicks on the thumbnail with the right handle trigger button, and the system pops up a floating panel 1.5m in front of the user, displaying the artwork snapshot and redrawing the creation process stroke by stroke at 33 frames per second. The user can freely move their head to view the drawing trajectory and overlay effects of each stroke from different angles.

[0058] It should be noted that, in this invention, although the specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for immersive digital media art display based on virtual reality, characterized in that, Includes the following steps: S1: Construct a virtual exhibition environment that includes a virtual showroom, color palette, and canvas; S2: Real-time acquisition of six-DOF pose information and button status information of the VR headset worn by the user and the controllers held in both hands; S3: Seamlessly switch between browsing mode and creation mode based on the grip button status of the right hand holding the controller; S4: In creation mode, the pose information of the right hand holding the controller is mapped to the pose information of the virtual brush. S5: Responds to the trigger button operation of the right hand holding the controller, picks up colors on the color palette and draws them in three-dimensional space; S6: Based on color picking, drawing, and brush strokes, complete the operation events, generate corresponding spatialized audio, and provide feedback to the user; S7: Save the brushstroke-level data during the drawing process and support playback of the creation process based on the brushstroke-level data.

2. The immersive digital media art display method based on virtual reality according to claim 1, characterized in that, In step S3, when it is detected that the grip button of the right hand controller is pressed continuously for more than a preset time, the system switches from browsing mode to creation mode; When the system detects that the grip button has been released, it automatically reverts to browsing mode.

3. The immersive digital media art display method based on virtual reality according to claim 1, characterized in that, In step S4, the position of the virtual pen tip is obtained by extending a preset distance in front of the right hand holding the controller, and the orientation of the virtual pen is consistent with the orientation of the right hand holding the controller.

4. The immersive digital media art display method based on virtual reality according to claim 1, characterized in that, In step S6, the generation of spatialized audio includes: The relative orientation is calculated based on the location of the sound source of the operation event and the user's head position, and the pre-stored audio samples are spatially rendered based on the head-related transfer function. The audio playback rate corresponding to the drawing operation is positively correlated with the pen stroke speed of the virtual brush.

5. The immersive digital media art display method based on virtual reality according to claim 1, characterized in that, In step S7, the brushstroke-level data is saved in a point sequence format containing three-dimensional coordinates and color information; When the creation process is replayed, the process of generating each brushstroke is reproduced stroke by stroke in the original time sequence, and users can freely adjust the viewing angle during the replay.

6. An immersive digital media art display system based on virtual reality, characterized in that, include: VR headsets are used to present immersive virtual scenes to users; A dual-hand controller is used to collect the user's hand operation commands and six-degree-of-freedom pose information; A computer host is communicatively connected to both the VR headset and the dual handheld controllers. The computer host runs a software system, which includes: The Display Elements Building Module is used to construct a virtual display environment that includes a virtual exhibition hall, color palette, and canvas. The data acquisition module is used to collect the posture information and button status information of the VR headset and the two handheld controllers in real time. The mode switching module is used to seamlessly switch between browsing mode and creation mode based on the gripping state of the right hand holding the controller. The brush mapping module is used to map the pose information of the right hand holding the controller to the pose information of the virtual brush in creation mode. The interactive drawing module is used to respond to the trigger button operation of the right hand holding the controller, pick up colors on the color palette and draw in three-dimensional space; The audio generation module is used to generate corresponding spatial audio based on color picking, drawing, and brush strokes to complete operation events and provide feedback to the user through the VR headset. The save and playback module is used to save the brushstroke-level data during the drawing process and supports the playback of the creation process based on the brushstroke-level data.

7. The immersive digital media art display system based on virtual reality according to claim 6, characterized in that, The mode switching module is specifically used for: When the right hand holding the controller's grip button is detected to be pressed continuously for more than a preset time, the control system switches from browsing mode to creation mode; When the grip button is detected to be released, the control system automatically returns to browsing mode.

8. The immersive digital media art display system based on virtual reality according to claim 6, characterized in that, The brush mapping module is specifically used for: Calculate the forward direction vector of the right-hand controller, and extend the position of the right-hand controller along the forward direction vector by a preset distance to obtain the position of the virtual pen tip. The orientation of the virtual pen is consistent with the orientation of the right-hand controller.

9. The immersive digital media art display system based on virtual reality according to claim 6, characterized in that, The audio generation module is specifically used for: The relative orientation is calculated based on the location of the sound source of the operation event and the user's head position, and the pre-stored audio samples are spatially rendered based on the head-related transfer function. The audio playback rate corresponding to the drawing operation is positively correlated with the pen stroke speed of the virtual brush.

10. The immersive digital media art display system based on virtual reality according to claim 6, characterized in that, The save and replay module is specifically used for: Save the stroke-level data in a point sequence format that includes three-dimensional coordinates and color information; When the creation process is replayed, the process of generating each brushstroke is reproduced stroke by stroke in the original time sequence, and users can freely adjust the viewing angle during the replay.

Images