Immersive head-mounted display-oriented real-time light field three-dimensional rendering method

Abstract

The invention discloses a real-time light field three-dimensional rendering method for an immersive head-mounted display, and belongs to the technical field of light field rendering, and the method comprises the steps: carrying out the uniform sampling of an angle space of original light field data, converting the original data into a YUV color space, and generating a multi-view image pseudo video sequence through zigzag scanning; an H.264 video encoder is adopted to compress the image pseudo video sequence, and the compressed image pseudo video sequence is uploaded to a server; the client side adopts WebGL and Three.js to decompress the light field data, and a three-dimensional texture map is generated; designing a vertex shader and a fragment shader; a plane in a three-dimensional space serves as proxy geometry of a scene, geometric data serves as input of a rasterization rendering pipeline, a vertex shader and a fragment shader are adopted, and a binocular viewpoint image is rendered in combination with three-dimensional mapping. By adopting the method, the real-time efficiency of light field three-dimensional rendering is improved, and the requirements of an immersive head-mounted display are met.

Description

Technical Field

[0001] The present invention relates to the technical field of light field rendering, and in particular to a real-time light field stereoscopic rendering method for an immersive head mounted display. Background Art

[0002] Light field rendering is an advanced computer graphics technology that simulates the behavior of light in the real world to generate realistic three-dimensional scenes. Its core is to capture and simulate the transmission of light in three-dimensional space, including direction, intensity and color. Traditional light field rendering technology research mainly focuses on view synthesis from a single viewpoint, using graphics rendering algorithms and neural network models to calculate or infer the image of the target viewpoint. In order to generate high-quality images, these technologies often rely on high-performance computing resources or even customized computing platforms to display new views, which not only increases costs, but also has poor real-time rendering performance, limiting the widespread application of light field technology, especially light field stereo rendering in immersive environments.

[0003] Therefore, it is necessary to provide a light field stereo rendering method in an immersive environment. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time light field stereo rendering method for an immersive head mounted display, thereby improving the real-time efficiency of light field rendering.

[0005] To achieve the above object, the present invention provides a real-time light field stereoscopic rendering method for an immersive head mounted display, the steps comprising:

[0006] S1, uniformly sampling the original light field data in the angle space, transforming the original data into the YUV color space, and using zigzag scanning to generate a multi-viewpoint image pseudo video sequence;

[0007] S2, compressing the image pseudo video sequence using the H.264 video encoder and uploading it to the server;

[0008] S3, loading the light field data transmitted to the server through the client, and decompressing the light field data using WebGL and Three.js to generate a three-dimensional texture map;

[0009] S4. Design a vertex shader and a fragment shader, wherein the fragment shader is designed based on a refocusing algorithm in a spatial domain;

[0010] S5. Use the plane in the three-dimensional space as the proxy geometry of the scene, use the geometric data as the input of the rasterization rendering pipeline, use the vertex shader and the fragment shader, and combine the three-dimensional map to render the binocular viewpoint image.

[0011] Preferably, the functions of the vertex shader in step S4 include clipping vertex coordinates and calculating the starting point and direction of light, and the formula for clipping vertex coordinates includes:

[0012] V clip =M projection *M view *M model *V;

[0013] Where V = (x, y, z, 1) represents the local coordinates of the vertex, V clip =(x, y, z, w) represents the converted clipping coordinates, M projection Represents the projection matrix of the virtual viewpoint camera, which is defined by the field of view angle and the near and far planes of projection. view Represents the view matrix, which is used to convert the world coordinates of the three-dimensional plane into camera coordinates. model Represents the model matrix, which is used to transform the local coordinates of the three-dimensional plane into world coordinates;

[0014] The starting point of the light is calculated according to the texture coordinates of the three-dimensional plane, and the direction of the light is calculated according to the position of the virtual camera and the local vertex coordinates of the three-dimensional plane. The formula is:

[0015] r o =[u,v];

[0016] r d =normalize(p camera -p plane );

[0017] In the formula, r d represents the direction of the light, [u,v] represents the texture coordinates of the three-dimensional plane, r o represents the starting point of the light, p camera represents the position of the virtual camera, p plane Represents the local vertex coordinates of a 3D plane.

[0018] Preferably, the fragment shader designed based on the spatial domain refocusing algorithm in step S4 includes:

[0019]

[0020] Where i, j represent the coordinates of the imaging plane, M 1 M 2 represents the sensor size, a represents the aperture size measurement, f represents the focal length measurement, F(x,y) represents the color value [R,G,B] of the light field sub-aperture image at the pixel coordinate x at the y coordinate, d represents the street distance vector between the current pixel and the target pixel, x screen ,y screen Represents the horizontal and vertical coordinates of screen pixels, dx , d y Respectively represent the horizontal and vertical components of the distance vector d.

[0021] Preferably, step S5 comprises:

[0022] S51, using a plane in the three-dimensional space as a geometric proxy for the scene, and setting the size of the plane according to the angular resolution of the light field;

[0023] S52, input the plane in the three-dimensional space into the rasterization rendering pipeline, use the vertex shader to clip the vertex coordinates, and calculate the direction of the light;

[0024] S53, converting the clipped coordinates into normalized device coordinates through perspective division;

[0025] S54, using a fragment shader to convert the device coordinates into final screen coordinates according to the current posture of the virtual camera, and generating a target view according to the aperture and focal length parameters of the refocusing;

[0026] S55: Rendering a binocular viewpoint image according to the position of the binocular viewpoint.

[0027] Preferably, in step S53, the clipped coordinates are converted into normalized device coordinates by perspective division, and the formula is:

[0028]

[0029] Where V NDC Represents device coordinates.

[0030] Preferably, the formula for converting the device coordinates to the final screen coordinates in step S54 is:

[0031]

[0032] Where V screen Represents screen coordinates, width and height represent the width and height of the viewport respectively, x offset and offset Indicates the offset of the lower left corner of the viewport on the screen.

[0033] Therefore, the present invention adopts the above-mentioned real-time light field stereo rendering method for immersive head-mounted display, which has the following beneficial effects:

[0034] (1) By controlling binocular rendering parameters based on a spatial domain refocusing algorithm, the scene depth, virtual camera focal length, and aperture can be controlled in an immersive head-mounted display, providing high-quality visual experience and interactivity.

[0035] (2) Improved the real-time efficiency of light field stereo rendering to meet the needs of immersive head-mounted displays;

[0036] (3) The light field stereo rendering of the present invention can run on low-end and mid-range computers and has good cross-platform performance. It only requires a browser that supports WebGL, which reduces the requirements of traditional light field computing on high-performance platforms.

[0037] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of a refocusing algorithm according to an embodiment of the present invention;

[0040] Figure 3 This is a binocular structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] Example

[0043] Reference Figure 1 The present invention provides a real-time light field stereoscopic rendering method for an immersive head mounted display, the steps comprising:

[0044] S1. Uniformly sample the original light field data in the angular space, transform the original data into the YUV color space, and use zigzag scanning to generate a multi-viewpoint image pseudo video sequence.

[0045] The original light field data refers to the scene image acquired by the light field data acquisition device such as the array camera. The main encoding methods are: multi-view image sequence and macro pixel image. The multi-view image can be taken by the camera array located in the two-dimensional plane, and each image is a traditional perspective projection image; unlike the multi-view image, the macro pixel image encodes the angle information of the light field in the pixel, that is, the macro pixel block is used to represent the projection of the same point in the three-dimensional space under different camera perspectives, which means that there is a fixed conversion relationship between the multi-view image and the macro pixel image. Assuming that s and t represent the pixel coordinates of each sub-view in the multi-view image, and u and v represent the position coordinates of the camera corresponding to the sub-view in the multi-view image on the two-dimensional plane, each pixel block of the macro pixel image can be represented as fixed s and t coordinates, and u*v pixels are extracted from the multi-view image. Conversely, if a macro pixel image is given, it is only necessary to traverse each pixel block of the image and extract pixels from the same coordinates s and t of each pixel block to generate a multi-view image.

[0046] S2. Use H.264 video encoder to compress the image pseudo video sequence and upload it to the server. Based on H.264 video encoder, the compression rate can reach 98.76%. H.264 video codec is an international standard widely used in video compression and network transmission. It was jointly developed by the International Telecommunication Union (ITU-T) and the International Organization for Standardization (ISO) to provide high compression efficiency while maintaining video quality.

[0047] S3, load the light field data transmitted to the server through the client, and use WebGL and Three.js to decompress the light field data to generate a three-dimensional texture map. WebGL and Three.js library: WebGL is a technology for rendering interactive three-dimensional graphics in a web browser. Three.js is a JavaScript library based on WebGL for creating and displaying three-dimensional graphics.

[0048] S4. Design vertex shaders and fragment shaders. The vertex shader processes the data of each vertex, including transforming the vertex position to the clipping space, applying lighting, etc. The fragment shading stage mainly uses the fragment shader to process the color and depth value of each pixel, and apply texture, lighting, and blending effects.

[0049] The functions of the vertex shader include clipping vertex coordinates and calculating the starting point and direction of light. The formula for clipping vertex coordinates includes:

[0050] V clip =M projection *M view *M model *V (1)

[0051] Where V = (x, y, z, 1) represents the local coordinates of the vertex, V clip =(x, y, z, w) represents the converted clipping coordinates, M projection Represents the projection matrix of the virtual viewpoint camera, which is defined by the field of view angle and the near and far planes of projection. view Represents the view matrix, which is used to convert the world coordinates of the three-dimensional plane into camera coordinates. model Represents the model matrix, which is used to transform the local coordinates of the three-dimensional plane into world coordinates;

[0052] The starting point of the light is calculated according to the texture coordinates of the three-dimensional plane, and the direction of the light is calculated according to the position of the virtual camera and the local vertex coordinates of the three-dimensional plane. The formula is:

[0053] r o =[u,v] (2)

[0054] r d =normalize(p camera -p plane ) (3)

[0055] In the formula, r d represents the direction of the light, [u,v] represents the texture coordinates of the three-dimensional plane, r o represents the starting point of the light, p camera represents the position of the virtual camera, p plane Represents the local vertex coordinates of a 3D plane.

[0056] The fragment shader is designed based on a refocusing algorithm in the spatial domain.

[0057] Reference Figure 2 , describes the basic principle model of refocusing. Take the two-dimensional projection of the light field, by changing the focal length F to F′ as an example. Given a refocused image I(x, y), it can be obtained by the sub-aperture image L of the light field F (u,v) is calculated according to formula (3).

[0058] I(x,y)=∫∫L F (u(α-1)+x,v(α-1)+y)dudv (3)

[0059] Where α is the refocusing factor, which represents the ratio of the original focal plane depth to the new focal plane depth. In order to facilitate actual image calculation, the uv coordinates are converted to the xy coordinates, and the new refocusing formula can be expressed as formula (4).

[0060]

[0061] Where ij is the sensor plane coordinate, kl is the microlens array coordinate, M1 M 2 is the sensor size, k∈[1,N 1 ],l∈[1,N 2 ],N 1 N 2 is the size of the microlens array, S ij (k, l) represents the sub-aperture image extracted by the ij coordinate. Therefore, the refocusing algorithm formula based on the spatial domain is as follows:

[0062]

[0063] Where i, j represent the coordinates of the imaging plane, M 1 M 2 represents the sensor size, a represents the aperture size measurement, f represents the focal length measurement, F(x,y) represents the color value [R,G,B] of the light field sub-aperture image at the pixel coordinate x at the y coordinate, d represents the street distance vector between the current pixel and the target pixel, x screen ,y screen Represents the horizontal and vertical coordinates of screen pixels, d x , d y Respectively represent the horizontal and vertical components of the distance vector d.

[0064] S5. Use the plane in the three-dimensional space as the proxy geometry of the scene, use the geometric data as the input of the rasterization rendering pipeline, use the vertex shader and the fragment shader, and combine the three-dimensional texture to render the binocular viewpoint image. Specifically include:

[0065] S51. Use a plane in the three-dimensional space as a geometric proxy for the scene, and set the size of the plane according to the angular resolution of the light field.

[0066] S52, input the plane in the three-dimensional space into the rasterization rendering pipeline, use formula (1) in the vertex shader to clip the vertex coordinates, and use formulas (2) and (3) to calculate the direction of the light. The range of the texture coordinates is [0,1], the texture coordinates of the geometric center of the plane are (0.5,0.5), and the texture coordinates of the four corner vertices are (0,0), (1,0), (1,1), (0,1), so as to construct the light parameter (r o ,r d ).

[0067] S53, convert the clipped coordinates into normalized device coordinates through perspective division of formula (7), the formula is:

[0068]

[0069] Where V NDC Represents device coordinates.

[0070] S54, using equations (5) and (6) in the fragment shader to convert the device coordinates into the final screen coordinates according to the current posture of the virtual camera, and generating a target view according to the refocusing aperture and focal length parameters.

[0071] When converting device coordinates to the final screen coordinates, since the range of NDC is usually [-1, 1], the viewport transformation of formula (8) is used to convert the NDC coordinates to the final screen coordinates. The formula is:

[0072]

[0073] Where V screen Represents screen coordinates, width and height represent the width and height of the viewport respectively, x offset and offset Indicates the offset of the lower left corner of the viewport on the screen.

[0074] S55: Rendering a binocular viewpoint image according to the position of the binocular viewpoint.

[0075] After steps S51-S54, a single viewpoint image can be rendered. However, the immersive head mounted display has binocular characteristics and needs to provide views for the left and right eyes respectively to produce a stereoscopic immersive feeling. Therefore, the above method is extended to the scene of binocular stereo rendering. In each rendering frame, a viewport and a rendering matrix are set for each view (corresponding to the left and right eyes). The binocular camera structure is as follows: Figure 3 shown.

[0076] For each frame, the left and right cameras are rendered twice according to the refocusing rendering algorithm to generate left and right eye images with horizontal parallax, so that the left and right eyes of the viewer can see two images at the same time, thus creating a binocular stereoscopic effect. The distance between the left and right cameras is set to the average interpupillary distance (IPD) of the human eye to generate a stereoscopic view that conforms to the human eye.

[0077] In order to verify the effectiveness of the method of the present invention, the present invention is compared with the traditional light field rendering method. Specifically, the network transmission delay of the present invention is 43ms on average, while that of the existing technology is 669ms, which is about 20 times higher than that of the existing technology. At the same time, the rendering time of each frame is 0.7ms on average, which is much higher than the 90HZ requirement of the immersive head-mounted display and meets the real-time requirement of VR content.

[0078] Therefore, the present invention adopts the above-mentioned real-time light field stereoscopic rendering method for immersive head-mounted display, which improves the real-time efficiency of light field stereoscopic rendering and meets the requirements of immersive head-mounted display.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A real-time light field stereo rendering method for an immersive head mounted display, characterized in that the steps include: S1, uniformly sampling the original light field data in the angle space, transforming the original data into the YUV color space, and using zigzag scanning to generate a multi-viewpoint image pseudo video sequence; S2, compressing the image pseudo video sequence using H.264 video encoder and uploading it to the server; S3, loading the light field data transmitted to the server through the client, and decompressing the light field data using WebGL and Three.js to generate a three-dimensional texture map; S4. Design a vertex shader and a fragment shader, wherein the fragment shader is designed based on a refocusing algorithm in a spatial domain; S5. Use the plane in the three-dimensional space as the proxy geometry of the scene, use the geometric data as the input of the rasterization rendering pipeline, use the vertex shader and the fragment shader, and combine the three-dimensional map to render the binocular viewpoint image.

2. A real-time light field stereo rendering method for an immersive head mounted display according to claim 1, characterized in that: The vertex shader in step S4 includes clipping vertex coordinates and calculating the starting point and direction of light. The clipping vertex coordinate formula is include: In clip =M projection *M view *M model *V; Where V = (x, y, z, 1) represents the local coordinates of the vertex, V clip =(x, y, z, w) represents the converted clipping coordinates, M projection Represents the projection matrix of the virtual viewpoint camera, which is defined by the field of view angle and the near and far planes of projection. view Represents the view matrix, which is used to convert the world coordinates of the three-dimensional plane into camera coordinates. model Represents the model matrix, which is used to transform the local coordinates of the three-dimensional plane into world coordinates; The starting point of the light is calculated according to the texture coordinates of the three-dimensional plane, and the direction of the light is calculated according to the position of the virtual camera and the local vertex coordinates of the three-dimensional plane. The formula is: r o =[u,v]; r d =normalize(p camera -p plane ); In the formula, r d represents the direction of the light, [u,v] represents the texture coordinates of the three-dimensional plane, r o represents the starting point of the light, p camera represents the position of the virtual camera, p plane Represents the local vertex coordinates of a 3D plane.

3. The real-time light field stereo rendering method for an immersive head mounted display according to claim 2, characterized in that: The fragment shader designed based on the spatial domain refocusing algorithm in step S4 includes: Where i, j represent the coordinates of the imaging plane, M1M2 represents the sensor size, a represents the aperture size measurement, f represents the focal length measurement, F(x,y) represents the color value [R, G, B] of the light field sub-aperture image at the pixel coordinate x at the y coordinate, d represents the street distance vector between the current pixel and the target pixel, and x screen ,y screen Represents the horizontal and vertical coordinates of screen pixels, d x , d y Respectively represent the horizontal and vertical components of the distance vector d.

4. The real-time light field stereo rendering method for an immersive head mounted display according to claim 3, characterized in that: Step S5 includes: S51, using a plane in the three-dimensional space as a geometric proxy for the scene, and setting the size of the plane according to the angular resolution of the light field; S52, input the plane in the three-dimensional space into the rasterization rendering pipeline, use the vertex shader to clip the vertex coordinates, and calculate the direction of the light; S53, converting the clipped coordinates into normalized device coordinates through perspective division; S54, using a fragment shader to convert the device coordinates into final screen coordinates according to the current posture of the virtual camera, and generating a target view according to the aperture and focal length parameters of the refocusing; S55: Rendering a binocular viewpoint image according to the position of the binocular viewpoint.

5. The real-time light field stereo rendering method for an immersive head mounted display according to claim 4, characterized in that: Step S53 converts the clipped coordinates into normalized device coordinates by perspective division: Where V NDC Represents device coordinates.

6. A real-time light field stereo rendering method for an immersive head mounted display according to claim 5, characterized in that: The formula for converting the device coordinates to the final screen coordinates in step S54 is: Where V screen Represents screen coordinates, width and height represent the width and height of the viewport respectively, x offset and offset Indicates the offset of the lower left corner of the viewport on the screen.

Images