A neural network-based XR head-mounted display video perspective method and system

Abstract

This invention relates to a neural network-based video perspective method and system for XR headsets, belonging to the field of image processing technology, and solves the problem that existing video perspective methods cannot reconstruct high-quality images in real time. The method includes acquiring binocular images captured in real time by an external binocular camera of the XR headset; after preprocessing, obtaining left and right camera depth maps by calculating disparity maps; acquiring colored point cloud data based on the preprocessed binocular images, left and right camera depth maps; projecting and meshing the colored point cloud data to obtain left and right eye composite images, which are then fed into a view optimization model to output optimized left and right eye images; the view optimization model is an improved lightweight conditional adversarial generative neural network; rendering the optimized left and right eye images and outputting them to the display terminal of the XR headset. This achieves real-time reconstruction of high-quality images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for XR head-mounted display video perspective based on neural networks. Background Technology

[0002] Extended Reality (XR) headsets utilize external camera modules and video passthrough technology to reconstruct environmental images in real time that match the user's real-world perspective. However, the inherent positional discrepancy between the external stereo camera and the user's eyes prevents precise alignment. This results in noticeable artifacts and scale inconsistencies in the reconstructed images using existing video passthrough techniques. The core reason for this is the lack of depth information, which becomes a critical bottleneck limiting the reconstruction effectiveness.

[0003] Existing perspective techniques include Passthrough+ and MPI-based (Multiplane-Image based) synthesis. Among them, Passthrough+ offers better real-time performance, while MPI-based supports multi-view synthesis.

[0004] However, Passthrough+ has coarse depth estimation and low image quality (low resolution, monochrome); while MPI-based methods have high computational complexity, cannot be processed in real time, and cannot meet the needs of online real-time synthesis of video perspective images. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a neural network-based XR head-mounted display video perspective method and system to solve the problem that existing video perspective methods cannot reconstruct high-quality images in real time.

[0006] On one hand, embodiments of the present invention provide a neural network-based XR head-mounted display video perspective method, comprising the following steps:

[0007] The system acquires binocular images captured in real time by the external binocular camera of the XR headset. After preprocessing, it calculates the disparity map to obtain the depth map of the left camera and the depth map of the right camera.

[0008] Based on the preprocessed binocular images, the depth map of the left camera, and the depth map of the right camera, colored point cloud data is obtained; the colored point cloud data is then subjected to left and right eye image projection and meshing to obtain the left and right eye composite images.

[0009] The left and right human eye synthesized images are fed into the view optimization model, which outputs the left and right human eye optimized images; the view optimization model is an improved lightweight conditional adversarial generative neural network.

[0010] The optimized images for the left and right eyes are rendered and output to the display terminal of the XR headset.

[0011] Based on further improvements to the above method, colored point cloud data is obtained from the preprocessed binocular images, the left camera depth map, and the right camera depth map, including:

[0012] Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the depth map of the left camera and the depth map of the right camera are calculated respectively, and the color of each pixel is obtained from the preprocessed binocular image and assigned to the corresponding 3D point cloud.

[0013] The coordinates of the 3D point cloud are unified to the same coordinate system based on the camera parameters to obtain colored point cloud data.

[0014] A further improvement to the above method involves projecting colored point cloud data onto left and right eye images. This is done based on the XR device calibration parameters and camera parameters, using the following formula to project the colored point cloud data onto the left and right eye images respectively, obtaining left and right eye coordinate data. The associated left and right eye color data are then stored separately.

[0015] ,

[0016] in, This represents the coordinate data of the left eye. This represents the coordinate data of the right eye. This represents colored point cloud data in the left camera coordinate system. This indicates the inverse operation; and These are the XR device calibration parameters, representing the pose of the left camera relative to the left eye and the pose of the right camera relative to the right eye, respectively. It is the camera calibration parameter in the camera parameters, which represents the pose of the right camera relative to the left camera.

[0017] Further improvements to the above method include gridded allocation, including:

[0018] After determining the size of the human eye image based on the camera parameters, the human eye image is divided according to the preset grid size to obtain a human eye image grid map;

[0019] Calculate the positions of the left and right eye coordinate data in the grid of the human eye image;

[0020] The color mean of each grid is calculated based on the color data of the left eye associated with the coordinate data of the left eye and the color data of the right eye associated with the coordinate data of the right eye.

[0021] Set the color of the grid without any coordinate data assignment to black.

[0022] Based on further improvements to the above method, the view optimization model includes: a generator and a discriminator; wherein, the generator is a multi-scale fully convolutional neural network built on the UNet network structure, and the discriminator is a multi-layer convolutional neural network built on the CNN network structure.

[0023] Based on the above method, the view optimization model optimizes the model parameters by constructing a pair of simulated image sets and using a loss function for supervision. Each pair of images in the pair of simulated image sets includes a conditional image and a corresponding target image. The conditional images are simulated composite images of the left and right eyes obtained by the method of acquiring the left and right eye composite images. The target images are simulated real images of the left and right eyes obtained by the trained image synthesis model based on the camera pose when the XR device acquires binocular images. The loss functions include adversarial loss, pixel-level loss, and structural similarity loss.

[0024] Based on further improvements to the above method, the loss function of the view optimization model is calculated using the following formula:

[0025] ,

[0026] in, This represents the loss function of the view optimization model. , and These represent adversarial loss, pixel-level loss, and structural similarity loss, respectively. and represents the weight factors for pixel-level loss and structural similarity loss, respectively; G and D represent the generator and discriminator in the view optimization model, respectively; x represents the input conditional image, and y represents the target image; This represents the image generated by the generator based on the input conditional image. This indicates that the discriminator distinguishes between real image pairs. The output discriminant value, This indicates that the discriminator generates image pairs. Output the discriminant value; This indicates the calculation of the mean; This indicates the calculation of the L1 norm. This represents a function that measures the structural similarity between two images.

[0027] Based on further improvements to the above method, and using the camera pose acquired by the XR device, simulated real images of the left and right eyes are obtained using a trained image synthesis model, including:

[0028] Using the left camera as the origin of the coordinate system, the pose of the left camera in the world coordinate system is obtained based on the time when the XR device acquires binocular images.

[0029] Based on the pose of the left camera in the world coordinate system, coordinate system transformation is performed using XR device calibration parameters and camera parameters to obtain the pose of the left human eye in the world coordinate system, as well as the pose of the right human eye in the world coordinate system; then, together with the corresponding binocular images, they are input into the trained image synthesis model to obtain the simulated real images of the left and right human eyes.

[0030] Based on further improvements to the above methods, the image synthesis model is obtained by training a 3DGS model or a NeRF model using binocular images, camera pose data, SLAM point cloud data, and camera parameters acquired by XR devices.

[0031] On the other hand, embodiments of the present invention provide a neural network-based XR head-mounted display video perspective system, comprising:

[0032] The binocular depth estimation module is used to acquire binocular images captured in real time by the external binocular camera of the XR headset. After preprocessing, the left camera depth map and the right camera depth map are obtained by calculating the disparity map.

[0033] The human eye image synthesis module is used to obtain colored point cloud data based on the preprocessed binocular image, the depth map of the left camera, and the depth map of the right camera; and to perform left and right human eye image projection and gridding on the colored point cloud data to obtain the left human eye synthesized image and the right human eye synthesized image.

[0034] The human eye image optimization module inputs the synthesized images of the left and right human eyes into the view optimization model and outputs the optimized images of the left and right human eyes; the view optimization model is an improved lightweight conditional adversarial generative neural network.

[0035] The image rendering and display module is used to render the optimized images for the left and right eyes and output them to the display terminal of the XR headset.

[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0037] 1. By using an external binocular camera to restore environmental depth and physical scale, the problem of inconsistent image content scale is effectively solved. By constructing an improved view optimization model of a conditional adversarial generative neural network, the human eye synthesized image is optimized, effectively solving the artifact problem in existing human eye synthesized images and improving image quality.

[0038] 2. Binocular depth maps are obtained through binocular depth estimation and then converted into colored point cloud data, expanding the data from a two-dimensional plane to a three-dimensional space. This includes multi-dimensional information such as the color, position, and depth of objects, enriching the content and details of the data. The point cloud data is projected onto the left and right human eye images to correct potential planar distortion or occlusion issues in the images, improving the overall accuracy of the data and more realistically restoring the three-dimensional structure of the scene and the spatial relationships of objects. The point cloud data is distributed using a human eye image mesh map to improve processing efficiency and obtain a meshed synthetic image. This image is then optimized using a view optimization model to obtain a high-precision human eye synthetic image, achieving a more intuitive and natural interaction method.

[0039] 3. No need to manually create complex 3D scene models; high-fidelity simulation training images can be obtained simply by training an image synthesis model.

[0040] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0041] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0042] Figure 1 This is a flowchart of a neural network-based XR head-mounted display video perspective method according to Embodiment 1 of the present invention;

[0043] Figure 2 This is a schematic diagram of a neural network-based XR head-mounted display video perspective system according to Embodiment 2 of the present invention. Detailed Implementation

[0044] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0045] Example 1

[0046] A specific embodiment of the present invention discloses a method for XR head-mounted display video perspective based on neural networks, such as... Figure 1 As shown, it includes the following steps:

[0047] S1. Acquire binocular images captured in real time by the external binocular camera of the XR headset. After preprocessing, obtain the depth map of the left camera and the depth map of the right camera by calculating the disparity map.

[0048] The XR headset features an external binocular camera, and the XR device calibration parameters and camera parameters are pre-acquired. The XR device calibration parameters include the pose of the left camera relative to the user's left eye. pose of the right camera relative to the right eye Pose includes both position and orientation matrices; camera parameters include: camera intrinsic parameter matrix (including camera focal length f, principal point coordinates of left and right cameras, etc.). and Camera distortion parameter D, camera calibration parameters (representing the pose of the right camera relative to the left camera), baseline distance B between the binocular cameras, image width w, and height h.

[0049] An external binocular camera captures binocular images at a certain frame rate (e.g., 60fps, 90fps, or 120fps). Therefore, in this embodiment, each frame of binocular image is processed and optimized before being output to the display terminal of the XR headset in real time. However, the processing procedure in this embodiment is described with one frame of binocular image as the processing object.

[0050] It should be noted that each frame of the stereo image includes a left camera image and a right camera image. The left and right camera images are preprocessed by performing stereo correction based on the camera parameters to eliminate image distortion, thereby transforming the original stereo image into an ideal parallel stereo image, which is convenient for subsequent stereo depth estimation.

[0051] For example, the cv2.stereoRectify() function provided by OpenCV is sampled and implemented. Its input includes camera parameters and the original stereo image, and the output is a stereo-corrected stereo image.

[0052] Furthermore, the left camera depth map and the right camera depth map are obtained by calculating the disparity map, including:

[0053] The RAFT-Stereo disparity estimation model was used to calculate the left disparity map of the left camera image and the right disparity map of the right camera image.

[0054] Based on camera parameters, the left and right camera depth maps corresponding to the left and right camera images are recovered using the left and right disparity maps, respectively.

[0055] It's important to note that a disparity map is a two-dimensional array where each element represents the horizontal displacement between corresponding pixels in the left and right camera images. For example, for each pixel in the left camera image, the value at the corresponding position in the left disparity map represents the horizontal offset of that pixel relative to its matching pixel in the right image. This value reflects the depth information of objects in the scene; the closer an object is to the camera, the larger the disparity value; the farther away an object is, the smaller the disparity value. In RAFT-Stereo, the disparity map is obtained through feature extraction, correlation calculation, and multi-level iterative updates on the input left and right images.

[0056] The depth maps of the left and right cameras are obtained by calculating the depth value of each pixel in the left and right disparity maps pixel by pixel using the following formulas:

[0057] ,

[0058] in, Represents pixels in the left / right disparity map The depth value, Represents pixels in the left / right disparity map The disparity value.

[0059] S2. Based on the preprocessed binocular images, the depth map of the left camera, and the depth map of the right camera, obtain colored point cloud data; perform left and right eye image projection and meshing on the colored point cloud data to obtain the left eye composite image and the right eye composite image.

[0060] It should be noted that, based on the preprocessed stereo images, the depth map from the left camera, and the depth map from the right camera, colored point cloud data is obtained, including:

[0061] Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the depth map of the left camera and the depth map of the right camera are calculated respectively, and the color of each pixel is obtained from the preprocessed binocular image and assigned to the corresponding 3D point cloud.

[0062] The coordinates of the 3D point cloud are unified to the same coordinate system based on the camera parameters to obtain colored point cloud data.

[0063] It is important to note that the depth maps from the left and right cameras have the same resolution as the preprocessed stereo images, and the depth values ​​at each coordinate correspond one-to-one with the corresponding RGB colors of the image. Based on the camera parameters, the coordinates of the 3D point cloud are calculated using the following formula based on the coordinates and depth values ​​of each pixel:

[0064] ,

[0065] in, Represents pixels The corresponding coordinates of the 3D point cloud P, and Each pixel Based on the 2D coordinates in the left or right camera depth map, the color value is obtained from the stereo image and assigned to the 3D point cloud P.

[0066] In this embodiment, the left camera is used as the origin of the coordinate system, and the parameters are calibrated using a stereo camera. The coordinates of the 3D point cloud corresponding to each pixel in the depth map of the right camera are unified to the coordinate system of the left camera, thereby obtaining all colored 3D point clouds in the coordinate system of the left camera.

[0067] Furthermore, the left and right eye image projections of the colored point cloud data are performed based on the XR device calibration parameters and camera parameters. The colored point cloud data is projected onto the left and right eye images respectively using the following formula to obtain the left eye coordinate data. And right eye coordinate data :

[0068] ,

[0069] in, This represents colored point cloud data in the left camera coordinate system. This indicates the inverse operation.

[0070] Simultaneously, left-eye color data associated with left-eye coordinate data and right-eye color data associated with right-eye coordinate data are stored separately; that is, the color data of the point cloud projected onto the left-eye image is stored in the left-eye color data. In the middle, a correlation is established with the corresponding left eye coordinate data, and the color data of the point cloud projected onto the right eye image is stored in the right eye color data. In the middle, a correlation is established with the corresponding right eye coordinate data.

[0071] For example, the left eye coordinate data is obtained by projecting point cloud P from a frame of colored point cloud data onto a left eye image. Then the color corresponding to point cloud P is the same as Establish a connection.

[0072] Furthermore, the left / right eye coordinate data and colors generated by the projection are gridded to obtain the composite image of the left eye and the composite image of the right eye, including:

[0073] After determining the size of the human eye image based on the camera parameters, the human eye image is divided according to a preset grid size to obtain a human eye image grid map; that is, the human eye image is the same size as the binocular image.

[0074] Calculate the grid positions assigned to the left and right eye coordinate data in the human eye image grid diagram;

[0075] Based on the left eye coordinate data and the right eye coordinate data, calculate the average color value of each assigned grid.

[0076] The color of the grid without any coordinate data assignment is set to black to obtain the composite image of the left and right eyes.

[0077] Specifically, in this embodiment, the preset grid size Given an image width of w and height of h in a predetermined camera parameter set to 2 pixels, what is the maximum grid position in the x-direction? Maximum grid position in the y direction , This indicates rounding down to the nearest integer.

[0078] The positions of the left and right eye coordinates in the human eye image grid are calculated using the following formula:

[0079] ,

[0080] in, It is one of the coordinate data of the left eye and the right eye. Mapped grid positions.

[0081] It should be noted that when , If the coordinates are set to 0, it indicates that either the left or right eye coordinates can be assigned to the human eye image grid. Otherwise, if the coordinates are outside the human eye image grid area, the corresponding left or right eye coordinates are discarded.

[0082] Because multiple eye coordinate data may be assigned to the same grid position, it is necessary to perform mean fusion processing on the grid colors. That is, the mean color is calculated based on the left eye color data associated with the left eye coordinate data assigned to each grid and the right eye color data associated with the right eye coordinate data.

[0083] Finally, by setting the color of the grid without any coordinate data assignment (i.e., the empty grid) to black, we obtain the composite image of the left and right eyes.

[0084] S3. Input the synthesized images of the left and right human eyes into the view optimization model, and output the optimized images of the left and right human eyes; the view optimization model is an improved lightweight conditional adversarial generative neural network.

[0085] It should be noted that artifacts may exist in the left and right human eye composite images obtained in step S2. Therefore, the view optimization model in this step is used to further optimize the artifacts in the composite images and reconstruct high-definition images.

[0086] The view optimization model in this embodiment is an improved lightweight conditional adversarial generative neural network, comprising: a generator and a discriminator; wherein, the generator is a multi-scale fully convolutional neural network built on the UNet network structure, which can take an image of arbitrary size as input; the discriminator is a multi-layer convolutional neural network built on the CNN network structure.

[0087] It should be noted that the generator includes an encoder module and a decoder module. The network structure is shown in Table 1. The encoder module gradually reduces the spatial size of the image through a series of convolutional and pooling layers (downsampling) while increasing the number of feature channels, thereby capturing the local and global features of the image. The decoder module is used to upsample through deconvolutional layers and concatenate the features extracted by the encoder, thereby gradually restoring the features to the same spatial size as the input image.

[0088] Specifically, the encoder module sequentially includes: a first feature mapping layer, a first multi-scale feature extraction layer, a first max pooling layer, a second multi-scale feature extraction layer, a second max pooling layer, a third multi-scale feature extraction layer, a feature normalization layer, and a second feature mapping layer. The first multi-scale feature extraction layer includes one multi-scale convolution operation and one ReLU activation function; the second multi-scale feature extraction layer includes two multi-scale convolution operations and two ReLU activation functions; and the third multi-scale feature extraction layer includes two multi-scale convolution operations and one ReLU activation function. The kernel sizes in the multi-scale convolution operations are 7×7, 5×5, and 3×3, with padding sizes of 3, 2, and 1, respectively, facilitating the capture of features at different scales.

[0089] The decoder module includes, in sequence, a second deconvolution feature recovery layer, a first deconvolution feature recovery layer, and an output layer; wherein, the second deconvolution feature recovery layer contains features concatenated from the second multi-scale feature extraction layer, and the first deconvolution feature recovery layer contains features concatenated from the first multi-scale feature extraction layer.

[0090] Table 1. Generator G Network Structure

[0091] ;

[0092] In Table 1, Conv represents convolution operation, MS Conv represents multi-scale convolution operation, BN represents BatchNorm, batch normalization processing, DeConv represents deconvolution operation, and Concat represents feature concatenation operation; Dropout(0.5) means randomly dropping some neurons with a probability of 0.5 to prevent overfitting and improve the generalization ability of the model.

[0093] It should be noted that the discriminator consists of a first convolutional layer, three second convolutional layers, and a fully connected layer. The network structure is shown in Table 2. The first convolutional layer includes convolution operations and the LeakyReLU activation function; the second convolutional layer includes convolution operations, batch normalization operations, and the LeakyReLU plan function; and the fully connected layer includes fully connected operations and the Sigmoid activation function.

[0094] Table 2 Discriminator D Network Structure

[0095] ;

[0096] In Table 2, FC represents a Fully Connected operation.

[0097] Before training the view optimization model, a set of paired simulated images is constructed as the training set. Each pair of images in the paired simulated image set includes a conditional image and a corresponding target image. The conditional images are the simulated composite images of the left and right eyes obtained using the method for acquiring the left and right eye composite images; that is, the binocular images acquired by the XR device for training are converted into simulated composite images of the left and right eyes according to steps S1 and S2. The target images are the simulated real images of the left and right eyes obtained using the trained image synthesis model based on the camera pose when the XR device acquired the binocular images.

[0098] It should be noted that the image synthesis model is obtained by training a 3DGS model or a NeRF model using stereo images, camera pose, SLAM point cloud data, and camera parameters acquired by an XR device. This method can obtain high-fidelity simulation training images without manually creating complex 3D scene models.

[0099] Furthermore, the simulated real images of the left and right eyes are obtained through the following steps:

[0100] ① Using the left camera as the origin of the coordinate system, the pose of the left camera in the world coordinate system is obtained according to the time when the XR device acquires the binocular images. In other words, a simulated composite image of the left and right human eyes is constructed as the conditional image based on each frame of binocular images acquired by the XR device. The simulated real image of the left and right human eyes is constructed as the target image based on the pose of the left camera in the world coordinate system at the corresponding time of each frame (calculated from the 6DOF pose of the XR device and the pose of the left camera relative to the XR device), thus forming a pair of images in the training set.

[0101] ② Based on the pose of the left camera in the world coordinate system, coordinate system transformation is performed using XR device calibration parameters and camera parameters to obtain the pose of the left human eye in the world coordinate system, as well as the pose of the right human eye in the world coordinate system; then, together with the corresponding binocular images, they are input into the trained image synthesis model to obtain the simulated real images of the left and right human eyes.

[0102] Specifically, based on the XR device calibration parameters and camera calibration parameters, the poses of the left and right eyes in the left camera coordinate system are calculated using the following formulas:

[0103] ,

[0104] in, This indicates the pose of the left eye relative to the left camera. This indicates the pose of the right eye relative to the left camera. and These are the XR equipment calibration parameters predetermined in step S1. These are the camera calibration parameters predetermined in step S1.

[0105] Based on the pose of the left camera in the world coordinate system, the pose of the left human eye in the world coordinate system and the pose of the right human eye in the world coordinate system are obtained using the following formulas:

[0106] ,

[0107] in, This represents the pose of the left eye in the world coordinate system. This represents the pose of the right eye in the world coordinate system. This represents the pose of the left camera in the world coordinate system. This formula is illustrated using a single frame of a stereo image, so the acquisition time for each frame is ignored.

[0108] Finally, , The images are fed into the trained image synthesis model along with the corresponding binocular images to obtain simulated real images of the left and right eyes.

[0109] During training, in the forward propagation process, each pair of images from the simulated image set is fed into the view optimization model. The generator receives the conditional image x (either the simulated composite image of the left eye or the simulated composite image of the right eye) from the image pair and outputs the generated image. Combine the conditional image x with the target image y (if x is a simulated composite image of the left human eye, then y is a simulated real image of the left human eye; if x is a simulated composite image of the right human eye, then y is a simulated real image of the right human eye) to form a real image pair. Then combine the conditional image x with the image generated by the generator. The discriminator receives both real and generated image pairs and predicts a probability value for each, representing the probability that the image pair is a real image pair. The generator aims to produce an image that closely resembles the target image based on the conditional image, maximizing the probability value output by the discriminator, thus successfully deceiving it. The discriminator's goal is to correctly distinguish between real and generated image pairs, outputting a high probability for real image pairs and a low probability for generated image pairs.

[0110] The loss function of the view optimization model includes: adversarial loss, pixel-level loss, and structural similarity loss. During backpropagation, the loss function is calculated to update the parameters of the discriminator, improving its ability to distinguish between real image pairs and generated image pairs; the parameters of the generator are also updated so that the generator can better "fool" the discriminator (through adversarial loss) and generate images that are closer to the target image (through pixel-level loss and structural similarity loss).

[0111] Specifically, the formula for the loss function is as follows:

[0112] ,

[0113] in, This represents the loss function of the view optimization model. , and These represent adversarial loss, pixel-level loss, and structural similarity loss, respectively. and represents the weight factors for pixel-level loss and structural similarity loss, respectively; G and D represent the generator and discriminator in the view optimization model, respectively; x represents the input conditional image, and y represents the target image; This represents the generated image output by the generator based on the input conditional image. This indicates that the discriminator distinguishes between real image pairs. The output discriminant value, This indicates that the discriminator generates image pairs. Output the discriminant value; This indicates the calculation of the mean; This indicates the calculation of the L1 norm. This represents a function that measures the structural similarity between two images.

[0114] Preferably, set Set to 0.01 It is 0.001.

[0115] It should be noted that the discriminator maximizes the adversarial loss to better distinguish between real and fake image pairs; while the generator attempts to minimize the adversarial loss in order to "deceive" the discriminator into believing that the generated image is real.

[0116] Pixel-level loss is used to measure the difference between the generated image and the real image, helping the generator to produce images that are closer to the real image at the pixel level, avoiding the generation of blurry images, and solving the problem of lack of detail in images caused by adversarial loss.

[0117] The Structural Similarity Index Measure (SSIM) loss is used to measure the structural similarity between generated and real images. It considers not only differences in pixel values ​​but also image brightness, contrast, and structural information, helping the generator produce images with correct texture and edge structure, thereby improving the overall image quality.

[0118] Preferably, the backpropagation of supervision information is achieved using the parameter optimizers provided by PyTorch (such as SGD, Adam, etc.), and the view optimization model is updated and iterated.

[0119] After training, the generator in the view optimization model is used for online real-time image optimization. That is, the synthesized images of the left and right eyes are fed into the generator in the view optimization model, and the optimized images of the left and right eyes are output.

[0120] S4. Render the optimized images for the left and right eyes and output them to the display terminal of the XR head-mounted display.

[0121] It should be noted that the display terminal of XR devices is usually a display screen, which is generally divided into left and right parts, corresponding to the user's left and right eyes respectively. This design makes it easier to achieve stereoscopic visual effects, allowing users to perceive depth and three-dimensional space.

[0122] In this step, 3D rendering engines such as OpenGL, Vulkan, and DirectX are used to render the left-eye optimized image and the right-eye optimized image, rendering two images for the left and right eyes respectively, and outputting them to the left and right display screens of the XR device respectively.

[0123] Compared with existing technologies, this embodiment provides a neural network-based XR head-mounted display video perspective method. By using an external binocular camera to recover environmental depth and physical scale, it effectively solves the problem of inconsistent image content scale. Through the construction of an improved conditional adversarial generative neural network view optimization model, it optimizes the synthesized human eye image, effectively solving the artifact problem present in existing synthesized human eye images and improving image quality. Binocular depth maps are obtained through binocular depth estimation and then converted into colored point cloud data, expanding the data from a two-dimensional plane to three-dimensional space. This includes multi-dimensional information such as the color, position, and depth of objects, enriching the data content and details. The point cloud data is projected onto the left and right human eye images to correct potential planar distortion or occlusion problems in the images, improving the overall data accuracy and more realistically restoring the three-dimensional structure of the scene and the spatial relationships of objects. The point cloud data is allocated using a human eye image mesh map to improve processing efficiency, resulting in a meshed synthesized image. This is then optimized using the view optimization model to obtain a high-precision synthesized human eye image, achieving a more intuitive and natural interaction method. High-fidelity simulation training images can be obtained simply by training an image synthesis model, without the need to manually create complex 3D scene models.

[0124] Example 2

[0125] Another embodiment of the present invention discloses a neural network-based XR head-mounted display video perspective system, thereby implementing the neural network-based XR head-mounted display video perspective method in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. Figure 2 As shown, the system includes:

[0126] The binocular depth estimation module 101 is used to acquire binocular images captured in real time by the external binocular camera of the XR head-mounted display. After preprocessing, the left camera depth map and the right camera depth map are obtained by calculating the disparity map.

[0127] The human eye image synthesis module 102 is used to obtain colored point cloud data based on the preprocessed binocular image, the depth map of the left camera and the depth map of the right camera; and to perform left and right human eye image projection and gridding on the colored point cloud data to obtain the left human eye synthesized image and the right human eye synthesized image.

[0128] The human eye image optimization module 103 inputs the left human eye synthesized image and the right human eye synthesized image into the view optimization model and outputs the left human eye optimized image and the right human eye optimized image; the view optimization model is an improved lightweight conditional adversarial generative neural network.

[0129] The image rendering and display module 104 is used to render the optimized images for the left and right eyes and output them to the display terminal of the XR head-mounted display.

[0130] Since the XR head-mounted display video perspective system based on neural networks in this embodiment and the aforementioned XR head-mounted display video perspective method based on neural networks can be mutually referenced, this description is redundant and will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.

[0131] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0132] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for XR head-mounted display video perspective based on neural networks, characterized in that, Includes the following steps: The system acquires binocular images captured in real time by the external binocular camera of the XR headset. After preprocessing, it calculates the disparity map to obtain the depth map of the left camera and the depth map of the right camera. Based on the preprocessed stereo images, the depth map from the left camera, and the depth map from the right camera, colored point cloud data is obtained. The colored point cloud data is subjected to left and right human eye image projection and gridding to obtain a left human eye composite image and a right human eye composite image; The left and right human eye composite images are input into the view optimization model, and the left and right human eye optimized images are output. The view optimization model is an improved lightweight conditional adversarial generative neural network. The optimized images for the left and right eyes are rendered and output to the display terminal of the XR head-mounted display. The step of projecting the colored point cloud data into left and right eye images involves projecting the colored point cloud data onto the left and right eye images respectively, based on the XR device calibration parameters and camera parameters, using the following formula to obtain left and right eye coordinate data, and storing the associated left and right eye color data respectively: ， in, This represents the coordinate data of the left eye. This represents the coordinate data of the right eye. This represents colored point cloud data in the left camera coordinate system. This indicates the inverse operation; and These are the XR device calibration parameters, representing the pose of the left camera relative to the left eye and the pose of the right camera relative to the right eye, respectively. It is the camera calibration parameter in the camera parameters, which represents the pose of the right camera relative to the left camera.

2. The XR head-mounted display video perspective method based on neural networks according to claim 1, characterized in that, The step of obtaining colored point cloud data based on the preprocessed binocular images, the left camera depth map, and the right camera depth map includes: Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the depth map of the left camera and the depth map of the right camera are calculated respectively, and the color of each pixel is obtained from the preprocessed binocular image and assigned to the corresponding 3D point cloud. The coordinates of the 3D point cloud are unified to the same coordinate system based on the camera parameters to obtain colored point cloud data.

3. The XR head-mounted display video perspective method based on neural networks according to claim 1 or 2, characterized in that, The gridded allocation includes: After determining the size of the human eye image based on the camera parameters, the human eye image is divided according to the preset grid size to obtain a human eye image grid map; Calculate the position of the left eye coordinate data and the right eye coordinate data in each grid of the human eye image grid diagram; The color mean of each grid is calculated based on the color data of the left eye associated with the coordinate data of the left eye and the color data of the right eye associated with the coordinate data of the right eye. Set the color of the grid without any coordinate data assignment to black.

4. The XR head-mounted display video perspective method based on neural networks according to claim 1, characterized in that, The view optimization model includes a generator and a discriminator; wherein the generator is a multi-scale fully convolutional neural network built on the UNet network structure, and the discriminator is a multi-layer convolutional neural network built on the CNN network structure.

5. The XR head-mounted display video perspective method based on neural networks according to claim 1 or 4, characterized in that, The view optimization model optimizes model parameters by constructing paired simulated image sets and using a loss function for supervision. Each pair of images in the paired simulated image set includes a conditional image and a corresponding target image. The conditional images are simulated composite images of the left and right eyes obtained by methods for acquiring left and right eye composite images. The target images are simulated real images of the left and right eyes obtained by using a trained image synthesis model based on the camera pose when the XR device acquires binocular images. The loss function includes adversarial loss, pixel-level loss, and structural similarity loss.

6. The XR head-mounted display video perspective method based on neural networks according to claim 5, characterized in that, The loss function of the view optimization model is calculated using the following formula: ， in, This represents the loss function of the view optimization model. , and These represent adversarial loss, pixel-level loss, and structural similarity loss, respectively. and represents the weight factors for pixel-level loss and structural similarity loss, respectively; G and D represent the generator and discriminator in the view optimization model, respectively; x represents the input conditional image, and y represents the target image; This represents the image generated by the generator based on the input conditional image. This indicates that the discriminator distinguishes between real image pairs. The output discriminant value, This indicates that the discriminator generates image pairs. Output the discriminant value; This indicates the calculation of the mean; This indicates the calculation of the L1 norm. This represents a function that measures the structural similarity between two images.

7. The XR head-mounted display video perspective method based on neural networks according to claim 5, characterized in that, The simulated real images of the left and right eyes, obtained using a trained image synthesis model based on the camera pose acquired by the XR device, include: Using the left camera as the origin of the coordinate system, the pose of the left camera in the world coordinate system is obtained based on the time when the XR device acquires binocular images. Based on the pose of the left camera in the world coordinate system, coordinate system transformation is performed using XR device calibration parameters and camera parameters to obtain the pose of the left human eye in the world coordinate system, as well as the pose of the right human eye in the world coordinate system; then, together with the corresponding binocular images, they are input into the trained image synthesis model to obtain the simulated real images of the left and right human eyes.

8. The XR head-mounted display video perspective method based on neural networks according to claim 5, characterized in that, The image synthesis model is obtained by training a 3DGS model or a NeRF model using binocular images, camera pose data, SLAM point cloud data, and camera parameters acquired by an XR device.

9. A neural network-based XR head-mounted display video perspective system, characterized in that, include: The binocular depth estimation module is used to acquire binocular images captured in real time by the external binocular camera of the XR headset. After preprocessing, the left camera depth map and the right camera depth map are obtained by calculating the disparity map. The human eye image synthesis module is used to obtain colored point cloud data based on the preprocessed binocular image, the depth map of the left camera, and the depth map of the right camera. The colored point cloud data is subjected to left and right human eye image projection and gridding to obtain a left human eye composite image and a right human eye composite image; The human eye image optimization module inputs the left human eye composite image and the right human eye composite image into the view optimization model, and outputs the left human eye optimized image and the right human eye optimized image; The view optimization model is an improved lightweight conditional adversarial generative neural network. The image rendering and display module is used to render the optimized image for the left human eye and the optimized image for the right human eye and output them to the display terminal of the XR head-mounted display; The step of projecting the colored point cloud data into left and right eye images involves projecting the colored point cloud data onto the left and right eye images respectively, based on the XR device calibration parameters and camera parameters, using the following formula to obtain left and right eye coordinate data, and storing the associated left and right eye color data respectively: ， in, This represents the coordinate data of the left eye. This represents the coordinate data of the right eye. This represents colored point cloud data in the left camera coordinate system. This indicates the inverse operation; and These are the XR device calibration parameters, representing the pose of the left camera relative to the left eye and the pose of the right camera relative to the right eye, respectively. It is the camera calibration parameter in the camera parameters, which represents the pose of the right camera relative to the left camera.

Images