Photo-realistic scene reconstruction and immersive interaction system for vr devices

Abstract

The application discloses a photo-level real scene reconstruction and immersive interaction system for a VR device. The application collects real environment images through a camera device of a user, obtains a Mesh model with semantic information and a corresponding texture feature image based on image training of a NeRF model, thereby reconstructing a three-dimensional scene model from two-dimensional images with high quality, and realizing seamless combination of a real environment and a virtual environment by using a Unity engine and a VR device, so as to provide an immersive visual experience for the user. In order to improve the interaction, the application innovatively adopts a mode of operation control according to user voice information, captures user voice instructions in real time through a microphone, and quickly and accurately converts the user voice instructions into corresponding text information to complete a series of semantic operations. The application realizes real-time and convenient three-dimensional scene reconstruction by fusing a NeRF model and VR technology, and realizes highly immersive interaction technology with user experience as the guide, so that the practicability of the system is pushed to a new height.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a photo-level real scene reconstruction and immersive interaction system for a VR device. BACKGROUND

[0002] The novel view synthesis task refers to inputting a given source image and a source pose, and synthesizing and rendering a target pose of any view angle. The neural radiance field optimizes the underlying continuous volume scene function by using a sparse set of input views, and can complete the novel view synthesis task of a complex scene with high quality, which is a milestone in three-dimensional reconstruction. However, the NeRF is low in efficiency when parameterizing the full connection neural network, and often needs 30s to render a frame of image, and real-time interaction requires scene reconstruction in a short time, so the prior art uses a grid to realize real-time novel view synthesis task.

[0003] The reconstructed three-dimensional model can only be viewed on a PC through screen display, and there is a certain physical distance between the user and the model when the user observes the model, and the visual immersion is low. And the user mainly interacts through the mouse and keyboard, such as translating, rotating and scaling the model, and the interactive operation is relatively limited, and cannot completely simulate the real-world operation.

[0004] One of the prior arts related to the present application is NeRF2Mesh, which is a work published by Peking University and Baidu on ICCV2023. The training of generating a grid model is divided into two stages: the first stage extracts a coarse grid based on the grid point representation of the neural radiation field, which is used for joint optimization of geometry and appearance; the second stage refines the coarse grid to optimize it into a fine grid with more accurate surface and adaptive surface density. NeRF2Mesh renders the final novel view image based on the extracted grid and feature map, and can realize real-time rendering on multiple platform devices using the support of traditional graphics pipeline for grid rendering. However, NeRF2Mesh cannot well estimate the normal vector of the object, which will make it difficult to capture appearance details in the case of fewer grid patches. In addition, although NeRF2Mesh can realize real-time rendering on multiple platform devices, for some low-performance devices such as mobile devices or embedded devices, its rendering efficiency may not be high enough, which limits the real-time performance. This makes NeRF2Mesh unable to provide a smooth user experience in some scenarios, limiting its wide application in mobile applications and virtual reality devices.

[0005] One of the prior art related to this invention is LERF, a work published at ICCV 2023 by a research team at UC Berkeley. This work proposes Language Embedding Radiation Fields (LLERFs), which embed language from off-the-shelf models such as CLIPs into NeRFs, enabling such open-ended language queries in 3D. LERF learns a dense, multi-scale language field within the NeRF by rendering CLIP embeddings along the training ray volume, supervising these embeddings in the training view to provide multi-view support. Figure One The underlying language field is smoothed and consistent. After optimization, LERF can interactively extract 3D relevance maps in real time for various language cues, showing great potential in robotics applications, visual language model understanding, and 3D scene interaction. However, for objects with similar semantic information, LERF may make errors in semantic queries and struggles to fully capture the spatial relationships of objects. During training, LERF constructs language fields by learning the CLIP embeddings corresponding to each cropped image, but this only reflects the semantic information of the image and cannot represent the relative positional relationship of that location in the entire space. Summary of the Invention

[0006] Existing 3D reconstruction technologies based on neural radiation fields are mostly limited to PC applications. However, they fail to meet the needs of 3D scene applications requiring high real-time interactivity, such as 3D games. Furthermore, existing NeRF models typically only generate density fields, lacking contextual or semantic information, thus limiting their ability to effectively interact with users. To address these technical challenges, this invention provides a photorealistic scene reconstruction and immersive interaction system for VR devices. It achieves breakthrough 3D scene reconstruction on VR devices and, combined with voice-text-semantic control, enables the system to accurately extract semantic information to locate objects and respond in real time, thereby possessing efficient real-time interaction capabilities and bringing users a completely new virtual experience.

[0007] The objective of this invention is achieved through the following technical solution: a photorealistic scene reconstruction and immersive interaction system for VR devices, comprising the following modules:

[0008] Image acquisition and training module: Users use camera equipment to acquire images of the real environment, use the acquired images as training views, train the images based on the NeRF model to obtain a Mesh model with semantic information and corresponding texture feature images, thereby reconstructing a three-dimensional scene model and / or object model from two-dimensional images with high quality;

[0009] a virtual environment real-time rendering module: loading the reconstructed three-dimensional scene model and / or object model through the Unity engine, the Unity engine being connected with the VR device and realizing real-time rendering, and the user visualizing the virtual scene and / or object rendered by the Unity engine through the VR device;

[0010] a voice control interaction module: detecting a keyword in an audio stream collected by a microphone of the user and converting the keyword into text information, parsing the text information into a semantic control instruction, and realizing control from text to semantics and corresponding perspective switching.

[0011] Further, the implementation of the image acquisition and training module is specifically as follows:

[0012] (1) using the NeRF2Mesh framework, learning the preliminary geometric structure through the density grid, and extracting the coarse grid from the density field by using the MarchingCubes algorithm;

[0013] (2) iteratively optimizing the coarse grid, including optimizing the vertex position, surface density and surface appearance, and using the differentiable rendering technology based on nvdiffrast to improve the geometric and appearance accuracy of the grid;

[0014] (3) using the color grid to simultaneously perform appearance learning in two stages, decomposing the appearance into view-independent diffuse color and view-dependent specular color; after the training is completed, the optimized fine grid, i.e., the optimized Mesh model, is exported, and its UV coordinates are unfolded, and the appearance is baked into a texture image;

[0015] (4) cleaning the exported optimized Mesh model of redundancy;

[0016] (5) rasterizing the cleaned Mesh model according to a given camera perspective to obtain the mapping coordinates corresponding to each pixel;

[0017] (6) using the mapping coordinates to calculate the difference between the rendered color and the true color through a fully connected neural network, and backward propagating the gradient to supervise the learning of the color feature, to obtain the preliminary position physical scale;

[0018] (7) based on the LERF framework, inputting the obtained position physical scale into the CLIP model to output a single CLIP vector; in the training process of embedding the NeRF2Mesh model in the CLIP model, a multi-scale feature pyramid constructed by the CLIP embedding generated by cropping from the training view is used for supervision, to obtain semantic information about the spatial position;

[0019] (8) Export the optimized mesh with semantic information in Mesh model, save the corresponding texture feature image, load the Mesh model and texture feature image into the virtual simulation platform, and realize the reconstruction of high-quality three-dimensional scene and / or object based on semantic information.

[0020] Further, the implementation of the virtual environment real-time rendering module is specifically:

[0021] (1) Configure the development environment of Unity engine to ensure that it can successfully connect with VR device;

[0022] (2) Import the Mesh model and corresponding texture feature image output by the image acquisition and training module, and configure the attributes and behavior characteristics of the reconstructed three-dimensional scene model and / or object model;

[0023] (3) According to the lighting setting in the scene, use Unity engine to calculate the lighting effect, including real-time shadow calculation, global illumination and real-time light source processing;

[0024] (4) Perform shading operation to apply the imported texture feature image to the Mesh model to form a map, so as to display the rendered object in the scene.

[0025] Further, the implementation of the voice control interaction module is specifically:

[0026] (1) Start the microphone, compress and encode the microphone audio stream, and push the audio stream to the cloud high-performance platform in real time;

[0027] (2) Continuously detect the input audio stream through the voice keyword detection model in the cloud high-performance platform, start voice-to-text conversion after detecting the keyword, convert the voice segments before and after the keyword into text information, and return to the local platform;

[0028] (3) The local platform preprocesses the received text information, removes the mood words and meaningless noise text, and sends it to the cloud language interaction model. The cloud language interaction model extracts the user operation purpose and converts the text information into specific user instructions;

[0029] (4) The local platform inputs the received user instructions into the Mesh model output by the image acquisition and training module, and combines the geometric information to frame the spatial range and coordinates of the specific object in the reconstructed scene generated by the model;

[0030] (5) Input the spatial coordinates into the virtual simulation platform, combine the current user virtual position, calculate the displacement curve, and realize the user perspective object focusing movement.

[0031] Further, creating a TCP connection between the Unity engine and the local platform includes the following steps:

[0032] a. Write a TCP Script to implement a TCP server in the Unity engine. This server can receive client connections and messages and process the received commands. The TCP Script consists of the following six parts: class and member variable definition, server startup, handling client connections, handling client messages, handling commands, and cleanup resources.

[0033] b. Create a new component under XR Origin in the Unity engine, import the TCP Script, and correctly set the IP address and port number in the Unity engine;

[0034] c. After running the Unity scene, the client and server will attempt to connect, and the client will receive a response from the server upon successful connection.

[0035] Furthermore, the TCP Script specifically refers to:

[0036] The class and member variable definition section is used to define a TCP class and its member variables, including the port number (port), the TCPListener object (server), and the camera object of the GameObject class.

[0037] In the server startup section, the StartServer method is used to start the server and the BeginAcceptTcpClient method is used to asynchronously accept client connections.

[0038] In handling the client connection part, first obtain the client's TcpClient object and initialize the NetworkStream, then use the BeginRead method to start asynchronously reading the data sent by the client;

[0039] In the client message processing section, the EndRead method is used to perform asynchronous reading, decode the received byte data and convert it into a string, and then the handleCommand method is called to process the received command;

[0040] The command processing section reads the object's output orientation and controls the camera's movement;

[0041] The resource cleanup section is used to stop the server and close client connections.

[0042] Furthermore, the system also has a handle control function, which allows users to freely move, observe, and operate the model in the virtual environment.

[0043] The beneficial effects of this invention are:

[0044] This invention captures real-world environmental images using the user's camera device, then converts them into high-quality 3D scenes in real time. Utilizing the Unity engine and VR devices, it seamlessly integrates the real and virtual environments, providing users with an immersive visual experience. To enhance interactivity, this invention innovatively employs a user-voice-based control method. The device's microphone captures the user's voice commands in real time and quickly and accurately converts them into corresponding text information, completing a series of semantic operations. Furthermore, based on the needs of some users, this invention also adds a gamepad control function to the system, further enhancing its practicality and interactive experience. In terms of innovation, this invention integrates the latest NeRF model and VR technology to implement real-time and convenient 3D scene reconstruction; and with a user-experience-oriented approach, it achieves highly immersive interactive technology, pushing the system's practicality to new heights. Whether used as an entertainment tool or applied in practical scenarios such as vocational training and online education, this system demonstrates enormous potential and broad application prospects. Attached Figure Description

[0045] Figure 1 This is a system architecture diagram provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the training process in the image acquisition and training module provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the interaction process in the voice control interaction module provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the TCP connection creation process provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] This invention provides a photorealistic scene reconstruction and immersive interaction system for VR devices. Users acquire images of the real environment using a camera, and a high-quality 3D scene is reconstructed from the 2D images by training a NeRF model. After optimization and iteration, mesh and texture information are exported. The 3D scene model and object models are loaded into the virtual environment using the Unity engine, and real-time rendering is achieved by connecting to a VR device. Users can activate the microphone on their VR device, and the system automatically detects keywords in the audio stream and converts them into text information, enabling control from text to semantics and corresponding viewpoint switching. Furthermore, considering the needs of some users, this invention adds a controller function, allowing users to freely move, observe, and manipulate models in the virtual environment, further enhancing interactivity.

[0052] The product form of this invention is an immersive visual media playback and interaction system for mixed reality, which can be ported to applications and software on multiple platforms and features high efficiency and real-time interactivity. The system architecture of this invention mainly includes three modules: an image acquisition and training module, a virtual environment real-time rendering module, and a voice control interaction module, such as... Figure 1 As shown below, the implementation process of each module is explained in detail.

[0053] I. Image Acquisition and Training Module: Users acquire images of the real environment using camera equipment. These acquired images are used as training views. Based on the NeRF model, the images are trained to obtain a Mesh model with semantic information and corresponding texture feature images, thereby reconstructing a high-quality 3D scene model and / or object model from 2D images; for example... Figure 2 As shown, the specific steps include:

[0054] 1) Using the NeRF2Mesh framework, the initial geometry is learned through density mesh, and the Marching Cubes algorithm is used to extract the coarse mesh from the density field.

[0055] 2) Iterative optimization of the coarse mesh, including vertex positions, face density, and surface appearance. Differentiable rendering techniques based on NVDiffrast are used to improve the geometric and appearance accuracy of the mesh.

[0056] 3) Utilize a color mesh to perform two-stage appearance learning simultaneously, decomposing the appearance into view-independent diffuse color and view-dependent specular color. After training, export the optimized fine mesh, i.e., the optimized Mesh model, and unfold its UV coordinates to bake the appearance into a texture image.

[0057] 4) Perform redundancy cleanup on the exported optimized Mesh model to ensure the efficiency of subsequent processing.

[0058] 5) Based on the given camera viewpoint, rasterize the cleaned Mesh model to obtain the texture coordinates corresponding to each pixel.

[0059] 6) Using a fully connected neural network, the difference between the rendered color and the ground truth color is calculated using the texture coordinates. The gradient is backpropagated to supervise the learning of color features and obtain the initial physical scale of the location.

[0060] 7) Based on the LERF framework, the obtained physical scale of location is input into the CLIP model, and a single CLIP vector is output. During the training process of the CLIP model embedding the NeRF2Mesh model, a multi-scale feature pyramid constructed from CLIP embeddings generated from the training view is used for supervision, and finally semantic information about spatial location is obtained.

[0061] 8) Export the optimized mesh with semantic information as a Mesh model and save the corresponding texture feature image. Load the Mesh model and texture feature image into the virtual simulation platform to realize the reconstruction of high-quality 3D scenes and / or objects based on semantic information.

[0062] Through the image acquisition and training module, users can easily acquire images of real-world environments and reconstruct high-quality 3D scenes using the NeRF model. This allows users to quickly obtain realistic virtual environments without complicated equipment and processes, saving time and costs.

[0063] II. Real-time Virtual Environment Rendering Module: This module loads the reconstructed 3D scene model and / or object model using the Unity engine. The Unity engine establishes a connection with the VR device and performs real-time rendering, allowing users to visualize the virtual scene and / or objects rendered by the Unity engine through their VR devices. Specifically, it includes the following steps:

[0064] 1) Configure the Unity engine development environment to ensure that it can successfully establish a connection with the VR device.

[0065] 2) Import the Mesh model output from the image acquisition and training module (format must be .obj), along with the corresponding texture feature image. Configure the attributes and behavioral characteristics of the reconstructed 3D scene model and / or object model.

[0066] 3) Based on the lighting settings in the scene, use the Unity engine to calculate the lighting effects, including real-time shadow calculation, global illumination, and real-time light source processing.

[0067] 4) Perform shading operations, apply the imported texture feature image to the Mesh model to form a texture map, and thus display the rendered object in the scene.

[0068] In one embodiment, the Pico all-in-one VR headset is selected to achieve a more realistic and immersive experience.

[0069] This invention utilizes the XR capabilities of the Unity engine, which encompasses Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies. By rapidly reconstructing imported mesh models, the virtual and real worlds can be effectively combined. The Unity engine boasts powerful cross-platform performance, allowing the developed XR applications to be seamlessly deployed to various devices, such as PCs, mobile devices, VR headsets, and AR glasses. Loading the reconstructed 3D scene and object models through the Unity engine ensures smooth scene rendering and interaction, resulting in a more fluid and natural user experience in the virtual environment, enhancing immersion and engagement.

[0070] III. Voice Control Interaction Module: The interaction environment is the terminal VR device. When the user activates the microphone on the terminal VR device, this module automatically detects keywords in the audio stream and converts them into text information. This text information is then parsed into semantic control commands, enabling control from text to semantics and corresponding viewpoint switching; for example... Figure 3 As shown, the specific steps include:

[0071] 1) Create a TCP connection between the Unity engine and the local platform, such as... Figure 4 As shown, the specific steps include:

[0072] a. Write a TCP Script, which mainly involves implementing a simple TCP server in the Unity engine. This server can receive client connections and messages and process the received commands. The TCP Script consists of six main parts: class and member variable definition, server startup, handling client connections, handling client messages, handling commands, and cleanup resources.

[0073] The class and member variable definition section is used to define a TCP class and its member variables, including the port number (port), the TCPListener object (server), and the camera object of the GameObject class, etc.

[0074] In the server startup section, the StartServer method is used to start the server and the BeginAcceptTcpClient method is used to asynchronously accept client connections.

[0075] In handling the client connection part, first obtain the client's TcpClient object and initialize the NetworkStream, then use the BeginRead method to start asynchronously reading the data sent by the client;

[0076] In the client message processing section, the EndRead method is used to perform asynchronous reading, decode the received byte data and convert it into a string, and then the handleCommand method is called to process the received command;

[0077] The command processing section reads the object's output orientation and controls the camera's movement;

[0078] The resource cleanup section is used to stop the server and close client connections.

[0079] b. Create a new component under XR Origin in the Unity engine, import the TCP Script, and correctly set the IP address and port number in the Unity engine.

[0080] c. After running the Unity scene, the client and server will attempt to connect. Once the connection is successful, a message "Hello, TCP Server!" will be sent and the server's response will be received.

[0081] 2) Start the microphone, compress and encode the microphone audio stream, and push the audio stream to the cloud high-computing platform in real time.

[0082] 3) The input audio stream is continuously detected by the voice keyword detection model on the cloud high-computing platform. After a keyword is detected, speech-to-text conversion is started. The speech segment from 3 seconds before the keyword is detected to 2 seconds after the volume is less than the threshold is converted into text information and sent back to the local platform.

[0083] 4) The local platform preprocesses the received text information, removes interjections and meaningless noise text, and then sends it to the cloud language interaction model. The cloud language interaction model extracts the user's operation purpose and converts the text information into specific user instructions, such as converting "Please focus your view on the table" into "focus table", and then sends it back to the local platform.

[0084] 5) The local platform inputs the received user commands into the Mesh model output by the image acquisition and training module. Combined with geometric information, it can select the spatial range and coordinates of specific objects in the reconstructed scene generated by the model.

[0085] 6) Input the spatial coordinates into the virtual simulation platform, combine them with the current virtual position of the user, calculate the displacement curve, and realize the focused movement of the object from the user's perspective.

[0086] The voice control and text conversion features allow users to control the virtual environment and switch perspectives using simple voice commands.

[0087] In addition, the system is equipped with a gamepad control function, which allows users to move, observe and manipulate models freely in the virtual environment. This enhances the user's interactivity and immersion, making the user experience more realistic and comfortable.

[0088] This implementation can extract 3D assets (scene geometry and semantics) of a scene from a set of input images with known camera parameters. Simultaneously, by extracting the 3D model of the scene into a surface rendering mesh, this implementation significantly reduces the rendering burden on mobile devices. Even on mobile devices with limited hardware resources, this invention can render in real time. Compared to existing technologies, it offers real-time interactivity and high efficiency, can train different scenes and complete reconstruction, can perform real-time semantic queries, and can respond quickly. This implementation also incorporates semantic information into scene interaction, enhancing scene interactivity. Introducing voice interaction for VR devices improves immersion; users can not only see objects in the virtual world but also adjust the viewpoint to a specific object using voice control, enhancing the interactive effect. This implementation establishes a TCP connection, solving the technical challenge of transmitting object output orientation to the Unity engine. This further improves the system's real-time performance and interactivity, providing users with a more stable and smooth experience. These effects have significant implications for 3D reconstruction, virtual tourism, and the gaming industry.

[0089] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A photorealistic scene reconstruction and immersive interaction system for VR devices, characterized in that, include: Image acquisition and training module: Users use camera equipment to acquire images of the real environment. These acquired images are used as training views. Based on the NeRF model, the images are trained to obtain a Mesh model with semantic information and corresponding texture feature images, thereby reconstructing a high-quality 3D scene model and / or object model from the 2D images. The specific implementation of the image acquisition and training module is as follows: (1) Using the NeRF2Mesh framework, the initial geometric structure is learned through density mesh, and the coarse mesh is extracted from the density field using the Marching Cubes algorithm; (2) Iterative optimization of coarse mesh, including optimization of vertex position, surface density and surface appearance, using differentiable rendering technology based on nvdiffrast to improve the geometric and appearance accuracy of mesh; (3) Use color mesh to perform appearance learning in two stages at the same time, decompose the appearance into view-independent diffuse color and view-dependent specular color; after training, export the optimized fine mesh, i.e. the optimized Mesh model, and expand its UV coordinates to bake the appearance into a texture image. (4) Perform redundancy cleanup on the exported optimized Mesh model; (5) Based on the given camera viewpoint, rasterize the cleaned Mesh model and obtain the texture coordinates corresponding to each pixel; (6) By using a fully connected neural network, the difference between the rendered color and the ground truth color is calculated using the texture coordinates. The gradient is backpropagated to supervise the learning of color features and obtain the initial physical scale of the location. (7) Based on the LERF framework, the obtained physical scale of the location is input into the CLIP model and a single CLIP vector is output. During the training process of the CLIP model embedding the NeRF2Mesh model, the multi-scale feature pyramid constructed from the CLIP embedding generated from the training view is used for supervision to obtain semantic information about the spatial location. (8) Export the optimized mesh with semantic information as a Mesh model and save the corresponding texture feature image. Load the Mesh model and texture feature image into the virtual simulation platform to realize the reconstruction of high-quality 3D scenes and / or objects based on semantic information. Virtual environment real-time rendering module: Loads the reconstructed 3D scene model and / or object model through the Unity engine, establishes a connection between the Unity engine and the VR device and realizes real-time rendering, and allows users to visualize the virtual scene and / or objects rendered by the Unity engine through the VR device; Voice control interaction module: Detects keywords in the audio stream captured by the user's microphone and converts them into text information. It then parses the text information into semantic control commands, enabling control from text to semantics and corresponding perspective switching.

2. The photorealistic scene reconstruction and immersive interaction system for VR devices according to claim 1, characterized in that, The implementation of the virtual environment real-time rendering module is as follows: (1) Configure the Unity engine development environment to ensure that it can successfully establish a connection with the VR device; (2) Import the Mesh model and corresponding texture feature image output by the image acquisition and training module, and configure the attributes and behavior characteristics of the reconstructed 3D scene model and / or object model; (3) Based on the lighting settings in the scene, use the Unity engine to calculate the lighting effects, including real-time shadow calculation, global illumination and real-time light source processing; (4) Perform a coloring operation, apply the imported texture feature image to the Mesh model to form a texture map, and then display the rendered object in the scene.

3. The photorealistic scene reconstruction and immersive interaction system for VR devices according to claim 1, characterized in that, The voice control interaction module is implemented as follows: (1) Start the microphone, compress and encode the microphone audio stream, and push the audio stream to the cloud high-computing platform in real time; (2) The input audio stream is continuously detected by the speech keyword detection model on the cloud high computing power platform. After the keyword is detected, speech to text is started, and the speech segments before and after the keyword are detected are converted into text information and sent back to the local platform. (3) The local platform preprocesses the received text information, removes interjections and meaningless noise text, and then sends it to the cloud language interaction model. The cloud language interaction model extracts the user's operation purpose and converts the text information into specific user instructions. (4) The local platform inputs the received user instructions into the Mesh model output by the image acquisition and training module, and combines geometric information to select the spatial range and coordinates of specific objects in the reconstructed scene generated by the model. (5) Input the spatial coordinates into the virtual simulation platform, combine them with the current virtual position of the user, calculate the displacement curve, and realize the focusing movement of the object from the user's perspective.

4. The photorealistic scene reconstruction and immersive interaction system for VR devices according to claim 3, characterized in that, Creating a TCP connection between the Unity engine and the local platform involves the following steps: a. Write a TCP Script to implement a TCP server in the Unity engine. This server can receive client connections and messages and process the received commands. The TCP Script consists of the following six parts: class and member variable definition, server startup, handling client connections, handling client messages, handling commands, and cleanup resources. b. Create a new component under XR Origin in the Unity engine, import the TCP Script, and correctly set the IP address and port number in the Unity engine; c. After running the Unity scene, the client and server will attempt to connect, and the client will receive a response from the server upon successful connection.

5. The photorealistic scene reconstruction and immersive interaction system for VR devices according to claim 4, characterized in that, The TCP Script specifically refers to: The class and member variable definition section is used to define a TCP class and its member variables, including the port number (port), the TCPListener object (server), and the camera object of the GameObject class. In the server startup section, the StartServer method is used to start the server and the BeginAcceptTcpClient method is used to asynchronously accept client connections. In handling the client connection part, first obtain the client's TcpClient object and initialize the NetworkStream, then use the BeginRead method to start asynchronously reading the data sent by the client; In the client message processing section, the EndRead method is used to perform asynchronous reading, decode the received byte data and convert it into a string, and then the handleCommand method is called to process the received command; The command processing section reads the object's output orientation and controls the camera's movement; The resource cleanup section is used to stop the server and close client connections.

6. The photorealistic scene reconstruction and immersive interaction system for VR devices according to claim 1, characterized in that, The system also features a handle control function, allowing users to freely move, observe, and manipulate models within the virtual environment.

Images