A method and system for image synthesis in XR devices based on multiple keyframes

Abstract

This invention relates to an image synthesis method and system for XR devices based on multiple keyframes, belonging to the field of image processing technology. It solves the problems of artifacts and scale inconsistencies in images synthesized by existing XR devices. The method includes: acquiring colored point cloud data for each frame based on real-time binocular images and depth maps from the left and right cameras of the XR device, and projecting this data onto the left and right eye images to obtain left and right eye coordinate data for each frame; constructing a historical keyframe point cloud set based on each frame of binocular images and colored point cloud data; allocating the left and right eye coordinate data for each frame to an eye image grid map, and assigning color values ​​to the empty grids generated by the allocation based on the historical keyframe point cloud set to obtain the synthesized left and right eye images for each frame; and inputting the synthesized left and right eye images for each frame into a neural network model to output optimized left and right eye images for each frame. This significantly improves the accuracy and quality of the synthesized images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image synthesis method and system for XR devices based on multiple keyframes. Background Technology

[0002] XR (Extended Reality) devices use external camera modules to capture external images and synthesize virtual and real images to construct a virtual environment that matches the real-world perspective. However, the physical positional difference between the external camera and the user's eyes results in low overlap, making artifacts and scale inconsistencies common in image reconstruction. Optimizing image synthesis and improving depth information acquisition and processing capabilities will significantly enhance the immersiveness and usability of XR devices, providing users with a more realistic and natural interactive experience and promoting the widespread application of XR technology in gaming, education, and industry.

[0003] Existing image synthesis methods such as Passthrough+ have good real-time performance, but their ability to process depth information is limited, and they cannot effectively solve the problems of artifacts and scale inconsistencies caused by the misalignment between the external camera and the user's eye position.

[0004] While existing MPI-based methods support multi-view synthesis, they require a large amount of computational resources to process multi-planar images, resulting in poor real-time performance and making it difficult to meet the high requirements of accuracy and real-time performance in image synthesis. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide an XR device image synthesis method and system based on multiple keyframes, so as to solve the problems of artifacts and scale inconsistencies in images synthesized by existing XR devices.

[0006] On one hand, embodiments of the present invention provide an XR device image synthesis method based on multiple keyframes, comprising the following steps:

[0007] The system acquires binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, it calculates the disparity map to obtain the left camera depth map and the right camera depth map of each frame of the binocular image.

[0008] Based on each frame of binocular image and its left and right camera depth maps, obtain colored point cloud data for each frame and project it onto the left and right eye images to obtain the left and right eye coordinate data for each frame; construct a set of historical keyframe point clouds based on each frame of binocular image and each frame of colored point cloud data.

[0009] Construct a human eye image grid map, allocate the left and right human eye coordinate data of each frame to the human eye image grid map, assign color values ​​to the empty grid generated by the allocation according to the historical key frame point cloud set, and obtain the left human eye composite image and the right human eye composite image of each frame.

[0010] Each frame of the synthesized left-eye image and each frame of the synthesized right-eye image are fed into the neural network model, which outputs each frame of the optimized left-eye image and each frame of the optimized right-eye image.

[0011] Based on further improvements to the above method, a set of historical keyframe point clouds is constructed using each frame of binocular image and each frame of colored point cloud data, including:

[0012] The first keyframe point cloud is obtained based on the pose of the left camera in the world coordinate system and the colored point cloud data of the first frame, and is added to the historical keyframe point cloud set along with the corresponding timestamp.

[0013] Project the last keyframe point cloud added to the historical keyframe point cloud set onto the left eye image in the current frame's binocular image, and calculate the projection coverage.

[0014] When the projection coverage is less than the preset projection rate threshold, the current frame is taken as the new keyframe. The current keyframe point cloud is obtained based on the pose of the left camera in the world coordinate system and the colored point cloud data of the current frame. The corresponding timestamp is added to the historical keyframe point cloud set.

[0015] Based on the further improvement of the above method, the last keyframe point cloud added to the historical keyframe point cloud set is projected onto the left eye image of the current frame, and the projection coverage is calculated, including:

[0016] Based on the pose of the left camera in the world coordinate system in the current frame and the point cloud of the last keyframe, obtain the point cloud projected onto the left camera coordinate system, and use it as the projected point cloud.

[0017] Based on the camera parameters, obtain the coordinates of the projection points in the left eye image of the current frame in the projection point cloud, and remove the projection points outside the range of the left eye image and duplicate projection points based on the coordinates of the projection points to obtain the number of projection points.

[0018] The projection coverage is obtained based on the number of projection points and the total number of pixels in the left eye image of the current frame.

[0019] Based on the further improvement of the above method, when the set of historical keyframe point clouds reaches the maximum number, the keyframe point cloud with the earliest timestamp is deleted every time a new keyframe point cloud is added.

[0020] Based on further improvements to the above method, colored point cloud data for each frame is obtained from each binocular image and its left and right camera depth maps, including:

[0021] Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the left camera depth map and the right camera depth map of each frame of stereo image are calculated respectively, and the color of each pixel is obtained from the corresponding stereo image and assigned to the corresponding 3D point cloud.

[0022] The coordinates of the 3D point cloud are unified to the left camera coordinate system based on the camera parameters, resulting in colored point cloud data for each frame.

[0023] A further improvement to the above method involves projecting the colored point cloud data of each frame onto the left and right eye images to obtain the left and right eye coordinate data for each frame. This is done based on the XR device calibration parameters and camera parameters, using the following formula to project the colored point cloud data of each frame onto the left and right eye images respectively, obtaining the left and right eye coordinate data for each frame, and storing the associated left and right eye color data respectively:

[0024]

[0025] in, and These represent the coordinates of the left eye and the right eye in frame t, respectively. (P3Ds) L_cam_t Let represent the colored point cloud data of the t-th frame in the left camera coordinate system, (·) -1 This indicates the inverse operation; and These are the XR device calibration parameters, representing the pose of the left camera relative to the left eye and the pose of the right camera relative to the right eye, respectively. It is the camera calibration parameter in the camera parameters, which represents the pose of the right camera relative to the left camera.

[0026] Based on the above method, a human eye image mesh is constructed, and the left and right eye coordinate data of each frame are assigned to the human eye image mesh, including:

[0027] After determining the size of the human eye image based on the camera parameters, the human eye image is divided according to the preset grid size to obtain a human eye image grid map;

[0028] Calculate the position of each grid in the human eye image grid diagram for each frame of left and right eye coordinate data;

[0029] The average color value of each assigned grid is calculated based on the left eye color data associated with the left eye coordinate data of each frame, and the right eye color data associated with the right eye coordinate data of each frame.

[0030] Based on a further improvement to the above method, the empty grid generated is a grid to which no left or right eye coordinate data has been assigned in the human eye image grid map; the color of the empty grid generated is assigned according to the historical keyframe point cloud set, including:

[0031] Each keyframe point cloud in the historical keyframe point cloud set is sequentially projected onto the left and right human eye images to obtain the left and right human eye projection data of each keyframe and its associated left and right human eye color data; and the depth values ​​of each keyframe point cloud projected onto the left and right cameras are calculated.

[0032] Based on the left and right eye projection data of the keyframe assigned to each empty grid, obtain the corresponding left and right eye color data and depth value of the keyframe, and assign the color data corresponding to the minimum depth value to the empty grid.

[0033] The remaining empty grid cells are assigned the color black.

[0034] Based on the further improvement of the above method, the depth value of the point cloud projected onto the left and right cameras for each keyframe is calculated using the following formula:

[0035]

[0036] in, and These represent the point cloud KP3Ds of the nth keyframe. w_n The depth values ​​projected onto the left camera and the depth values ​​projected onto the right camera; {·}[2] indicates that the z-coordinate value is extracted. This represents the pose of the left camera in the world coordinate system at the time of the nth keyframe.

[0037] On the other hand, embodiments of the present invention provide an XR device image synthesis system based on multiple keyframes, comprising:

[0038] The binocular depth estimation module is used to acquire binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, the left camera depth map and right camera depth map of each frame of binocular image are obtained by calculating the disparity map.

[0039] The human eye image projection module is used to obtain colored point cloud data for each frame based on each binocular image and its left and right camera depth maps, and project it onto the left and right human eye images to obtain the left and right human eye coordinate data for each frame; and to construct a set of historical key frame point clouds based on each binocular image and each frame of colored point cloud data.

[0040] The human eye image synthesis module is used to construct a human eye image grid map, allocate the left and right human eye coordinate data of each frame to the human eye image grid map, call the historical key frame point cloud set to assign color values ​​to the empty grid generated by the allocation, and obtain the left human eye synthesized image and the right human eye synthesized image of each frame.

[0041] The human eye image optimization module inputs the left and right human eye composite images of each frame into the neural network model and outputs the left and right human eye optimized images of each frame.

[0042] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0043] 1. By using binocular depth estimation technology to obtain binocular depth maps, the physical scale is restored, effectively solving the problem of inconsistent image content scale. Then, the depth map is combined with the color image to convert it into colored point cloud data, which expands the data from a two-dimensional plane to a three-dimensional space, integrating multi-dimensional information such as the color, position, and depth of objects, greatly enriching the content and details of the data.

[0044] 2. By projecting colored point cloud data onto the left and right human eye images, potential planar distortions or occlusions in the images are corrected, improving data accuracy and more realistically restoring the 3D structure of the scene and the spatial relationships of objects. Simultaneously, historical keyframes are constructed. When allocating point cloud data using the human eye image mesh, historical keyframes are used to fill in empty areas, effectively reducing artifacts generated during image synthesis. Finally, a neural network model is used to optimize the data, resulting in a high-precision synthesized human eye image, enabling a more intuitive and natural interaction.

[0045] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0046] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0047] Figure 1 This is a flowchart of an XR device image synthesis method based on multiple keyframes according to Embodiment 1 of the present invention;

[0048] Figure 2 This is a schematic diagram of an XR device image synthesis system based on multiple keyframes in Embodiment 2 of the present invention. Detailed Implementation

[0049] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0050] Example 1

[0051] A specific embodiment of the present invention discloses an image synthesis method for XR devices based on multiple keyframes, such as... Figure 1 As shown, it includes the following steps:

[0052] S1. Acquire binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, obtain the left camera depth map and right camera depth map of each frame of binocular image by calculating the disparity map.

[0053] The XR device uses an external binocular camera, and the XR device calibration parameters and camera parameters are acquired in advance. The XR device calibration parameters include the pose of the left camera relative to the left eye of the user. pose of the right camera relative to the right eye Pose includes the position and orientation matrix; camera parameters include: camera intrinsic matrix (including camera focal length f, and the principal point coordinates of the left and right cameras are both c). x and c y Camera distortion parameter D, camera calibration parameters (representing the pose of the right camera relative to the left camera), baseline distance B between the two cameras, image width w, and height h.

[0054] An external binocular camera captures binocular images at a certain frame rate (e.g., 60fps, 90fps, or 120fps). In this embodiment, each frame of binocular image is processed and optimized before being output in real time to the display terminal of the XR device.

[0055] It should be noted that each frame of the stereo image includes both a left camera image and a right camera image. Preprocessing of the left and right camera images involves stereo correction based on camera parameters to eliminate image distortion, thereby transforming the original stereo image into an ideal parallel stereo image, which facilitates subsequent stereo depth estimation.

[0056] For example, the cv2.stereoRectify() function provided by OpenCV is sampled and implemented. Its input includes camera parameters and the original stereo image, and the output is a stereo-corrected stereo image.

[0057] Furthermore, the left and right camera depth maps for each frame of the stereo image are obtained by calculating the disparity map, including:

[0058] The RAFT-Stereo disparity estimation model was used to calculate the left disparity map of the left camera image and the right disparity map of the right camera image.

[0059] Based on camera parameters, the left and right camera depth maps corresponding to the left and right camera images are recovered using the left and right disparity maps, respectively.

[0060] It's important to note that a disparity map is a two-dimensional array where each element represents the horizontal displacement between corresponding pixels in the left and right camera images. For example, for each pixel in the left camera image, the value at the corresponding position in the left disparity map represents the horizontal offset of that pixel relative to its matching pixel in the right image. This value reflects the depth information of objects in the scene; the closer an object is to the camera, the larger the disparity value; the farther away an object is, the smaller the disparity value. In RAFT-Stereo, the disparity map is obtained through feature extraction, correlation calculation, and multi-level iterative updates on the input left and right images.

[0061] The depth maps of the left and right cameras are obtained by calculating the depth value of each pixel in the left and right disparity maps pixel by pixel using the following formulas:

[0062]

[0063] Where, depth p d represents the depth value of pixel p in the left / right disparity map. p This represents the disparity value of pixel p in the left / right disparity map.

[0064] S2. Based on each frame of binocular image and its left and right camera depth maps, obtain the colored point cloud data for each frame and project it onto the left and right eye images to obtain the left and right eye coordinate data for each frame; based on each frame of binocular image and each frame of colored point cloud data, construct a set of historical keyframe point clouds.

[0065] It should be noted that, based on each frame of the stereo image and its left and right camera depth maps, colored point cloud data for each frame is obtained, including:

[0066] Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the left camera depth map and the right camera depth map of each frame of stereo image are calculated respectively, and the color of each pixel is obtained from the corresponding stereo image and assigned to the corresponding 3D point cloud.

[0067] The coordinates of the 3D point cloud are unified to the left camera coordinate system based on the camera parameters, resulting in colored point cloud data for each frame.

[0068] It is important to note that the depth maps from the left and right cameras have the same resolution as the preprocessed stereo images, and the depth values ​​at each coordinate correspond one-to-one with the corresponding RGB colors of the image. Based on the camera parameters, the coordinates of the 3D point cloud are calculated using the following formula based on the coordinates and depth values ​​of each pixel:

[0069]

[0070] Among them, (X) p ,Y p Z p ) represents pixel p(p x ,p y The coordinates of the corresponding 3D point cloud P, p x and p y These are the 2D coordinates of pixel p in the depth map of the left camera or the depth map of the right camera. Based on these coordinates, the color is obtained from the stereo image and assigned to the 3D point cloud P.

[0071] In this embodiment, the left camera is used as the origin of the coordinate system, and the parameters are calibrated using a stereo camera. The coordinates of the 3D point cloud corresponding to each pixel in the depth map of the right camera corresponding to each frame of the stereo image are unified to the coordinate system of the left camera, so as to obtain all the colored 3D point clouds in each frame under the coordinate system of the left camera.

[0072] Furthermore, the colored point cloud data of each frame is projected onto the left and right eye images to obtain the left and right eye coordinate data for each frame. This is done based on the XR device calibration parameters and camera parameters, using the following formulas to project the colored point cloud data of each frame onto the left and right eye images respectively, thus obtaining the left eye coordinate data and the right eye coordinate data for each frame.

[0073]

[0074] in, and These represent the coordinates of the left eye and the right eye in frame t, respectively. (P3Ds) L_cam_t Let represent the colored point cloud data of the t-th frame in the left camera coordinate system, (·) -1 This indicates the inverse operation.

[0075] Simultaneously, the left eye color data associated with the left eye coordinate data of each frame and the right eye color data associated with the right eye coordinate data of each frame are stored separately. That is, the color data of the point cloud projected onto the left eye image is stored in the left eye color data. In the middle, a correlation is established with the corresponding left eye coordinate data, and the color data of the point cloud projected onto the right eye image is stored in the right eye color data. In the middle, a correlation is established with the corresponding right eye coordinate data.

[0076] For example, the left eye coordinate data l(l) is obtained by projecting the point cloud P in a frame of colored point cloud data onto the left eye image. x ,l y If the color of point cloud P is the same as that of l (l x ,l y Establish a connection.

[0077] Simultaneously, based on each frame of stereo image and each frame of colored point cloud data, a set of historical keyframe point clouds is constructed, including:

[0078] ① Obtain the first keyframe point cloud based on the pose of the left camera in the world coordinate system and the colored point cloud data of the first frame, and put it into the historical keyframe point cloud set along with the corresponding timestamp.

[0079] It should be noted that in this embodiment, the first frame of the stereo image is used as the first keyframe. The colored point cloud data of the first keyframe corresponding to the first keyframe is transformed according to the pose of the left camera in the world coordinate system to obtain the point cloud of the first keyframe in the world coordinate system. The formula is as follows:

[0080]

[0081] Among them, KP3Ds w_1 This represents the point cloud of the first keyframe. This represents the pose of the left camera in the world coordinate system at the moment of the first keyframe, calculated based on the 6DOF pose of the XR device and the pose of the left camera relative to the XR device; P3Ds L_cam_1 This represents the first frame of colored point cloud data.

[0082] ② Project the last keyframe point cloud added to the historical keyframe point cloud set onto the left eye image in the current frame binocular image, and calculate the projection coverage.

[0083] Specifically, it includes:

[0084] Based on the pose of the left camera in the world coordinate system in the current frame And the final keyframe point cloud KP3Ds w_last Obtain the point cloud P3Ds projected onto the left camera coordinate system. L_cam_cur As a projected point cloud, the formula is as follows:

[0085]

[0086] Based on the camera parameters, the coordinates of the projection point in the left eye image of the current frame in the projection point cloud are obtained using the following formula:

[0087]

[0088] in, and These represent the x and y coordinates of the projection point, respectively, in P3Ds. L_cam_cur [0]、P3Ds L_cam_cur [1] and P3Ds L_cam_cur [2] represents the x, y and z coordinates of the projected point cloud, respectively, and int(·) represents the rounding operation.

[0089] Furthermore, based on the width w and height h of the left eye image in the current frame, to and Given the condition, retain the projection points whose x and y coordinates are both within the range of the left eye image of the current frame, and after removing duplicate projection points with the same x and y coordinates, count the number of remaining projection points.

[0090] Furthermore, the projection coverage is obtained based on the number of projection points and the total number of pixels in the left eye image of the current frame.

[0091] ③ When the projection coverage is less than the preset projection rate threshold, the current frame is taken as the new key frame. The current key frame point cloud is obtained based on the pose of the left camera in the world coordinate system and the colored point cloud data of the current frame. The corresponding timestamp is added to the historical key frame point cloud set.

[0092] It should be noted that the historical keyframe point cloud set has a maximum number. When the historical keyframe point cloud set reaches the maximum number, the keyframe point cloud with the earliest timestamp is deleted every time a new keyframe point cloud is added.

[0093] S3. Construct a human eye image grid map, allocate the left and right human eye coordinate data of each frame to the human eye image grid map, assign color values ​​to the empty grid generated by the allocation according to the historical key frame point cloud set, and obtain the left human eye composite image and the right human eye composite image of each frame.

[0094] It should be noted that constructing the human eye image mesh involves assigning the left and right eye coordinate data of each frame to the human eye image mesh, including:

[0095] After determining the size of the human eye image based on the camera parameters, the human eye image is divided according to the preset grid size to obtain a human eye image grid map;

[0096] Calculate the position of each grid in the human eye image grid diagram for each frame of left and right eye coordinate data;

[0097] The average color value of each assigned grid is calculated based on the left eye color data associated with the left eye coordinate data of each frame, and the right eye color data associated with the right eye coordinate data of each frame.

[0098] Specifically, in this embodiment, the preset grid size is 2 pixels, and the image width is w and the height is h in the predetermined camera parameters. Therefore, the maximum grid position in the x-direction is... Maximum grid position in the y direction This indicates rounding down to the nearest integer.

[0099] The positions of the left and right eye coordinates in each frame within the human eye image grid are calculated using the following formula:

[0100]

[0101] Among them, (grid) x ,grid y ) is one of the coordinate data (D) in the left eye coordinate data and the right eye coordinate data. x D y The grid position mapped.

[0102] It should be noted that when 0 ≤ grid x <Max x ,0≤grid y <Max y If the coordinates are set to 0, it indicates that either the left or right eye coordinates can be assigned to the human eye image grid. Otherwise, if the coordinates are outside the human eye image grid area, the corresponding left or right eye coordinates are discarded.

[0103] Because multiple eye coordinate data may be assigned to the same grid position, it is necessary to perform mean fusion processing on the grid colors. That is, the mean color is calculated based on the left eye color data associated with the left eye coordinate data assigned to each grid and the right eye color data associated with the right eye coordinate data.

[0104] Considering that empty grids may occur during grid allocation, meaning that no grid is assigned to any left or right eye coordinate data in the human eye image grid, the empty grids generated during allocation are color-assigned based on the historical keyframe point cloud set, including:

[0105] ① Project each keyframe point cloud in the historical keyframe point cloud set onto the left and right human eye images in sequence to obtain the left and right human eye projection data of each keyframe and its associated left and right human eye color data; and calculate the depth values ​​of each keyframe point cloud projected onto the left and right cameras.

[0106] It should be noted that projecting the point cloud of each keyframe onto the left and right eye images is similar to projecting the colored point cloud data of each frame onto the left and right eye images in step S2, as shown in the following formula:

[0107]

[0108] in, and These represent the left eye projection data and the right eye projection data of the nth keyframe, respectively. This represents the pose of the left camera in the world coordinate system at the time of the nth keyframe, in KP3Ds. w_n This represents the point cloud of the nth keyframe.

[0109] The color data of each keyframe point cloud projected onto the left eye image is stored in the corresponding keyframe left eye color data and associated with the corresponding keyframe left eye projection data; the color data of each keyframe point cloud projected onto the right eye image is stored in the corresponding keyframe right eye color data and associated with the corresponding keyframe right eye projection data.

[0110] Simultaneously, the point cloud of each keyframe is projected onto the left and right cameras, and the z-coordinate is extracted from the projection data as the depth value, as shown in the following formula:

[0111]

[0112] in, and represents the depth values ​​of the point cloud projected onto the left camera and the depth values ​​of the point cloud projected onto the right camera in the nth keyframe, respectively; {·}[2] represents the z-coordinate value extracted from the three-dimensional data.

[0113] ② Based on the left and right eye projection data of the keyframes assigned to each empty grid, obtain the corresponding left and right eye color data and depth values ​​of the keyframes, and assign the color data corresponding to the minimum depth value to the empty grid.

[0114] It should be noted that, following the method described above for allocating the left and right eye coordinate data of each frame to the human eye image grid, the left and right eye projection data of each keyframe are also allocated to the human eye image grid. However, based only on the left and right eye projection data of the keyframe allocated to each empty grid, the corresponding left and right eye color data and depth value of the keyframe are obtained, and the color data corresponding to the minimum depth value is assigned to the empty grid.

[0115] ③ Assign the color of the remaining empty grid to black (0,0,0).

[0116] This embodiment uses historical keyframes to fill in the hole areas, effectively reducing artifacts generated during image synthesis.

[0117] S4. Input the left eye synthesized image and the right eye synthesized image of each frame into the neural network model, and output the left eye optimized image and the right eye optimized image of each frame.

[0118] It should be noted that this step utilizes a neural network model to further optimize the synthesized left-eye image and the synthesized right-eye image for each frame, reconstructing a high-resolution image.

[0119] The neural network model in this embodiment is an improved lightweight conditional adversarial generative neural network, comprising: a generator and a discriminator; wherein, the generator is a multi-scale fully convolutional neural network built on the UNet network structure, which can take an image of arbitrary size as input; the discriminator is a multi-layer convolutional neural network built on the CNN network structure.

[0120] It should be noted that the generator includes an encoder module and a decoder module. The network structure is shown in Table 1. The encoder module gradually reduces the spatial size of the image through a series of convolutional and pooling layers (downsampling) while increasing the number of feature channels, thereby capturing the local and global features of the image. The decoder module is used to upsample through deconvolutional layers and concatenate the features extracted by the encoder, thereby gradually restoring the features to the same spatial size as the input image.

[0121] Specifically, the encoder module sequentially includes: a first feature mapping layer, a first multi-scale feature extraction layer, a first max pooling layer, a second multi-scale feature extraction layer, a second max pooling layer, a third multi-scale feature extraction layer, a feature normalization layer, and a second feature mapping layer. The first multi-scale feature extraction layer includes one multi-scale convolution operation and one ReLU activation function; the second multi-scale feature extraction layer includes two multi-scale convolution operations and two ReLU activation functions; and the third multi-scale feature extraction layer includes two multi-scale convolution operations and one ReLU activation function. The kernel sizes in the multi-scale convolution operations are 7×7, 5×5, and 3×3, with padding sizes of 3, 2, and 1, respectively, facilitating the capture of features at different scales.

[0122] The decoder module includes, in sequence, a second deconvolution feature recovery layer, a first deconvolution feature recovery layer, and an output layer; wherein, the second deconvolution feature recovery layer contains features concatenated from the second multi-scale feature extraction layer, and the first deconvolution feature recovery layer contains features concatenated from the first multi-scale feature extraction layer.

[0123] Table 1. Generator G Network Structure

[0124]

[0125]

[0126] In Table 1, Conv represents convolution operation, MS Conv represents multi-scale convolution operation, BN represents BatchNorm, batch normalization processing, DeConv represents deconvolution operation, and Concat represents feature concatenation operation; Dropout(0.5) means randomly dropping some neurons with a probability of 0.5 to prevent overfitting and improve the generalization ability of the model.

[0127] It should be noted that the discriminator consists of a first convolutional layer, three second convolutional layers, and a fully connected layer. The network structure is shown in Table 2. The first convolutional layer includes convolution operations and the LeakyReLU activation function; the second convolutional layer includes convolution operations, batch normalization operations, and the LeakyReLU plan function; and the fully connected layer includes fully connected operations and the Sigmoid activation function.

[0128] Table 2. Discriminator D Network Structure

[0129]

[0130]

[0131] In Table 2, FC represents a Fully Connected operation.

[0132] Before training the neural network model, a set of paired simulated images is constructed as the training set. Each pair of images in the paired simulated image set includes a conditional image and a corresponding target image. The conditional images are simulated composite images of the left and right eyes obtained using the method for acquiring the left and right eye composite images; that is, the binocular images acquired by the XR device for training are converted into simulated composite images of the left and right eyes according to steps S1-S3. The target images are simulated real images of the left and right eyes obtained using the trained image synthesis model based on the camera pose when the XR device acquired the binocular images.

[0133] It should be noted that the image synthesis model is obtained by training a 3DGS model or a NeRF model using stereo images, camera pose, SLAM point cloud data, and camera parameters acquired by an XR device. This method can obtain high-fidelity simulation training images without manually creating complex 3D scene models.

[0134] Furthermore, the simulated real images of the left and right eyes are obtained through the following steps:

[0135] ① Using the left camera as the origin of the coordinate system, the pose of the left camera in the world coordinate system for each frame of binocular image acquired by the XR device is obtained according to the time of acquisition. In other words, a simulated composite image of the left and right human eyes is constructed as the conditional image based on each frame of binocular image acquired by the XR device, and a simulated real image of the left and right human eyes is constructed as the target image based on the pose of the left camera in the world coordinate system at the corresponding time of each frame of image, thus forming a pair of images in the training set.

[0136] ② Based on the pose of the left camera in the world coordinate system for each frame, coordinate system transformation is performed using XR device calibration parameters and camera parameters to obtain the pose of the left human eye in the world coordinate system for each frame, as well as the pose of the right human eye in the world coordinate system for each frame; then, the pose is input into the trained image synthesis model to obtain the simulated real image of the left human eye and the simulated real image of the right human eye.

[0137] Specifically, based on the XR device calibration parameters and camera calibration parameters, the poses of the left and right eyes in the left camera coordinate system are calculated using the following formulas:

[0138]

[0139] in, This indicates the pose of the left eye relative to the left camera. This indicates the pose of the right eye relative to the left camera. and These are the XR equipment calibration parameters predetermined in step S1. These are the camera calibration parameters predetermined in step S1.

[0140] Based on the pose of the left camera in the world coordinate system for each frame, the pose of the left human eye in the world coordinate system for each frame, and the pose of the right human eye in the world coordinate system for each frame, are obtained using the following formulas:

[0141]

[0142] in, Let represent the pose of the left eye in the world coordinate system at frame t. Let represent the pose of the right eye in the world coordinate system at frame t. This represents the pose of the left camera in the world coordinate system at frame t.

[0143] Finally, and The images are fed into the trained image synthesis model along with the binocular image of frame t to obtain the simulated real image of the left eye and the simulated real image of the right eye.

[0144] During training, in the forward propagation process, each pair of simulated images is fed into the neural network model. The generator receives the conditional image x (either a simulated image of the left or right human eye) from the image pair and outputs the generated image G(x). The conditional image x is paired with the target image y (if x is a simulated image of the left human eye, then y is a simulated real image of the left human eye; if x is a simulated image of the right human eye, then y is a simulated real image of the right human eye) to form a real image pair. The conditional image x is paired with the image G(x) generated by the generator to form a generated image pair. The discriminator receives the real image pairs and the generated image pairs and predicts the probability values ​​for each, which represent the probability that the image pair is a real image pair. The generator's goal is to generate an image that is as close as possible to the target image based on the conditional image, maximizing the probability value output by the discriminator, thus successfully deceiving the discriminator. The discriminator's goal is to correctly distinguish between real and generated image pairs, i.e., outputting a high probability for real image pairs and a low probability for generated image pairs.

[0145] The loss functions of neural network models include: adversarial loss, pixel-level loss, and structural similarity loss. During backpropagation, the loss functions are calculated to update the parameters of the discriminator, improving its ability to distinguish between real image pairs and generated image pairs; the parameters of the generator are also updated, enabling the generator to both better "fool" the discriminator (through adversarial loss) and generate images that are closer to the target image (through pixel-level loss and structural similarity loss).

[0146] Specifically, the formula for the loss function is as follows:

[0147]

[0148] Where loss represents the loss function of the neural network model, L cGAN (G,D), L L1 (G) and L ssim (G) represents the adversarial loss, pixel-level loss, and structural similarity loss, respectively; λ1 and λ2 represent the weight factors of pixel-level loss and structural similarity loss, respectively; G and D represent the generator and discriminator in the neural network model, respectively; x represents the input conditional image, and y represents the target image; G(x) represents the generated image output by the generator based on the input conditional image, D(x,y) represents the discriminant value output by the discriminator for the real image pair (x,y), and D(x,G(x)) represents the discriminant value output by the discriminator for the generated image pair (x,G(x)); E[·] represents the calculation of the mean; ||·||1 represents the calculation of the L1 norm, and ssim(·,·) represents the function that measures the structural similarity between two images.

[0149] Preferably, λ1 is set to 0.01 and λ2 is set to 0.001.

[0150] It should be noted that the discriminator maximizes the adversarial loss to better distinguish between real and fake image pairs; while the generator attempts to minimize the adversarial loss in order to "deceive" the discriminator into believing that the generated image is real.

[0151] Pixel-level loss is used to measure the difference between the generated image and the real image, helping the generator to produce images that are closer to the real image at the pixel level, avoiding the generation of blurry images, and solving the problem of lack of detail in images caused by adversarial loss.

[0152] The Structural Similarity Index Measure (SSIM) loss is used to measure the structural similarity between generated and real images. It considers not only differences in pixel values ​​but also image brightness, contrast, and structural information, helping the generator produce images with correct texture and edge structure, thereby improving the overall image quality.

[0153] Preferably, the parameter optimizer provided by PyTorch (such as SGD, Adam, etc.) is used to implement the backpropagation of supervision information and to update and iterate the neural network model.

[0154] After training, the generator in the neural network model is used for online real-time image optimization. That is, the synthesized left eye image and the synthesized right eye image of each frame are fed into the generator in the neural network model, and the optimized left eye image and the optimized right eye image of each frame are output.

[0155] It should be noted that the display terminal of XR devices is usually a display screen, which is generally divided into left and right parts, corresponding to the user's left and right eyes respectively. This design makes it easier to achieve stereoscopic visual effects, allowing users to perceive depth and three-dimensional space.

[0156] Finally, using 3D rendering engines such as OpenGL, Vulkan, and DirectX, the optimized images for the left and right eyes are rendered, producing two images for the left and right eyes respectively, which are then output to the left and right display screens of the XR device.

[0157] Compared with existing technologies, this embodiment provides an XR device image synthesis method based on multiple keyframes. It utilizes binocular depth estimation technology to obtain a binocular depth map, restoring the physical scale and effectively solving the problem of inconsistent image content scale. The depth map is then combined with a color image to convert it into colored point cloud data, expanding the data from a two-dimensional plane to three-dimensional space. This integrates multi-dimensional information such as object color, position, and depth, greatly enriching the data content and detail. The colored point cloud data is projected onto the left and right eye images to correct potential planar distortion or occlusion issues, improving data accuracy and more realistically restoring the three-dimensional structure of the scene and the spatial relationships of objects. Simultaneously, historical keyframes are constructed. When allocating point cloud data using a human eye image mesh, these historical keyframes are used to fill in empty areas, effectively reducing artifacts generated during image synthesis. Finally, a neural network model is used to optimize the data, resulting in a high-precision synthesized human eye image, enabling a more intuitive and natural interaction.

[0158] Example 2

[0159] Another embodiment of the present invention discloses an XR device image compositing system based on multiple keyframes, thereby implementing the XR device image compositing method based on multiple keyframes in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. Figure 2 As shown, the system includes:

[0160] The binocular depth estimation module 101 is used to acquire binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, the left camera depth map and the right camera depth map of each frame of binocular image are obtained by calculating the disparity map.

[0161] The human eye image projection module 102 is used to obtain colored point cloud data for each frame based on each binocular image and its left and right camera depth maps, and project it onto the left and right human eye images to obtain the left and right human eye coordinate data for each frame; and to construct a set of historical key frame point clouds based on each binocular image and each frame of colored point cloud data.

[0162] The human eye image synthesis module 103 is used to construct a human eye image grid map, allocate the left and right human eye coordinate data of each frame to the human eye image grid map, call the historical key frame point cloud set to assign color values ​​to the empty grid generated by the allocation, and obtain the left human eye synthesized image and the right human eye synthesized image of each frame.

[0163] The human eye image optimization module 104 inputs the left human eye composite image and the right human eye composite image of each frame into the neural network model, and outputs the left human eye optimized image and the right human eye optimized image of each frame.

[0164] Since the XR device image synthesis system based on multiple keyframes in this embodiment and the aforementioned XR device image synthesis method based on multiple keyframes can be mutually referenced, and this is a repetition of the previous description, they will not be repeated here. Because this system embodiment shares the same principle as the above method embodiment, it also possesses the corresponding technical effects of the above method embodiment.

[0165] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0166] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image synthesis in XR devices based on multiple keyframes, characterized in that, Includes the following steps: The system acquires binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, it calculates the disparity map to obtain the left camera depth map and the right camera depth map of each frame of the binocular image. Based on each frame of the binocular image and its left and right camera depth maps, colored point cloud data for each frame is acquired and projected onto the left and right eye images to obtain the left and right eye coordinate data for each frame. This is done according to the XR device calibration parameters and camera parameters, using the following formula to project the colored point cloud data for each frame onto the left and right eye images respectively, obtaining the left and right eye coordinate data for each frame. The associated left and right eye color data are then stored separately. ， in, and They represent the first Frame left eye coordinate data and the first Frame right eye coordinate data, Indicates the first position in the left camera coordinate system Frame-by-frame colored point cloud data, This indicates the inverse operation; and These are the XR device calibration parameters, representing the pose of the left camera relative to the left eye and the pose of the right camera relative to the right eye, respectively. These are the camera calibration parameters in the camera parameters, representing the pose of the right camera relative to the left camera; based on each frame of binocular image and each frame of colored point cloud data, a set of historical keyframe point clouds is constructed. Constructing a human eye image grid map and allocating left and right eye coordinate data of each frame to the human eye image grid map includes: determining the size of the human eye image according to camera parameters, and dividing the human eye image according to a preset grid size to obtain the human eye image grid map; calculating the position of each grid allocated to the left and right eye coordinate data of each frame in the human eye image grid map; calculating the average color value of each allocated grid based on the left eye color data associated with the left eye coordinate data of each frame and the right eye color data associated with the right eye coordinate data of each frame; assigning color values ​​to the empty grids generated by the allocation according to the historical keyframe point cloud set to obtain the composite image of the left and right eyes of each frame, including: sequentially projecting each keyframe point cloud in the historical keyframe point cloud set onto the left and right eye images to obtain the left and right eye projection data of each keyframe and its associated keyframe left and right eye color data; and calculating the depth values ​​of each keyframe point cloud projected onto the left and right cameras using the following formula: ， in, and They represent the first Keyframe point cloud Depth values ​​projected onto the left camera and depth values ​​projected onto the right camera; Indicates taking out z Coordinate values Indicates the first The pose of the left camera in the world coordinate system at the time of each keyframe; the empty grid is a grid in the human eye image grid map that does not have any left or right human eye coordinate data assigned to it. Based on the left and right human eye projection data of the keyframe assigned to each empty grid, the corresponding left and right human eye color data and depth value of the keyframe are obtained, and the color data corresponding to the minimum depth value is assigned to the empty grid; the color of the remaining empty grid is assigned to black. Each frame of the synthesized left-eye image and each frame of the synthesized right-eye image are fed into the neural network model, which outputs each frame of the optimized left-eye image and each frame of the optimized right-eye image.

2. The XR device image synthesis method based on multiple keyframes according to claim 1, characterized in that, The step of constructing a historical keyframe point cloud set based on each frame of binocular image and each frame of colored point cloud data includes: The first keyframe point cloud is obtained based on the pose of the left camera in the world coordinate system and the colored point cloud data of the first frame, and is added to the historical keyframe point cloud set along with the corresponding timestamp. Project the last keyframe point cloud added to the historical keyframe point cloud set onto the left eye image in the current frame's binocular image, and calculate the projection coverage. When the projection coverage is less than the preset projection rate threshold, the current frame is taken as the new keyframe. The current keyframe point cloud is obtained based on the pose of the left camera in the world coordinate system and the colored point cloud data of the current frame. The corresponding timestamp is added to the historical keyframe point cloud set.

3. The XR device image synthesis method based on multiple keyframes according to claim 2, characterized in that, The step of projecting the last keyframe point cloud added to the historical keyframe point cloud set onto the left eye image of the current frame and calculating the projection coverage includes: Based on the pose of the left camera in the world coordinate system in the current frame and the point cloud of the last keyframe, obtain the point cloud projected onto the left camera coordinate system, and use it as the projected point cloud. Based on the camera parameters, obtain the coordinates of the projection points in the left eye image of the current frame in the projection point cloud, and remove the projection points outside the range of the left eye image and duplicate projection points based on the coordinates of the projection points to obtain the number of projection points. The projection coverage is obtained based on the number of projection points and the total number of pixels in the left eye image of the current frame.

4. The XR device image synthesis method based on multiple keyframes according to claim 2, characterized in that, When the set of historical keyframe point clouds reaches its maximum size, the keyframe point cloud with the earliest timestamp is deleted whenever a new keyframe point cloud is added.

5. The XR device image synthesis method based on multiple keyframes according to claim 1, characterized in that, The step of obtaining colored point cloud data for each frame based on each binocular image and its left and right camera depth maps includes: Based on the camera parameters, the coordinates of the 3D point cloud corresponding to each pixel in the left camera depth map and the right camera depth map of each frame of stereo image are calculated respectively, and the color of each pixel is obtained from the corresponding stereo image and assigned to the corresponding 3D point cloud. The coordinates of the 3D point cloud are unified to the left camera coordinate system based on the camera parameters, resulting in colored point cloud data for each frame.

6. A multi-keyframe-based XR device image compositing system, used to execute the multi-keyframe-based XR device image compositing method as described in any one of claims 1-5, characterized in that, include: The binocular depth estimation module is used to acquire binocular images captured in real time by the external binocular camera of the XR device. After preprocessing, the left camera depth map and right camera depth map of each frame of binocular image are obtained by calculating the disparity map. The human eye image projection module is used to obtain colored point cloud data for each frame based on each binocular image and its left and right camera depth maps, and project it onto the left and right human eye images to obtain the left and right human eye coordinate data for each frame. Based on each frame of stereo image and each frame of colored point cloud data, construct a set of historical keyframe point clouds; The human eye image synthesis module is used to construct a human eye image grid map, allocate the left and right human eye coordinate data of each frame to the human eye image grid map, call the historical key frame point cloud set to assign color values ​​to the empty grid generated by the allocation, and obtain the left human eye synthesized image and the right human eye synthesized image of each frame. The human eye image optimization module inputs the left and right human eye composite images of each frame into the neural network model and outputs the left and right human eye optimized images of each frame.

Images