A VR rendering method based on reconstructing three-dimensional spatial relationships of images

Abstract

The present invention proposes a VR rendering method based on reconstructing three-dimensional spatial relationships of images, which belongs to the field of virtual reality technology. The method comprises: formulating camera array rules according to rendering requirements; and setting according to the formulated camera array rules in three-dimensional software that needs to perform rendering; firstly, executing a rendering task in the three-dimensional software to render the image in the camera; after the rendering is completed, reconstructing the three-dimensional spatial relationship in the synthesis software, and establishing pixel fusion between images; finally, executing the rendering task in the synthesis software; the method decomposes a single rendering pressure into multiple rendering tasks; the number of decompositions can be changed according to requirements, the more the rendering tasks are decomposed, the higher the resolution that can be achieved under the same rendering conditions, and the smaller the rendering pressure that can be achieved under the same resolution, which solves the problem of serious hardware load for rendering ultra-high resolution under conventional GPU rendering processes, reduces rendering time, and improves production efficiency.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a VR rendering method based on reconstructing three-dimensional spatial relationships of images. Background Art

[0002] The standard VR format on the market is 2:1 Panorama. To render a 2:1 panorama in 3D software, you can only use the built-in panoramic rendering standard in the 3D software or a third-party external VR renderer. With the continuous development of the VR industry, the market demand for the clarity of VR videos and pictures has also increased, which has also brought huge challenges to VR ultra-high-definition rendering. At present, the traditional VR rendering method in the industry has the following pain points and difficulties: a) For the 3D engine under the GPU rendering module, it is impossible to perform block rendering when rendering VR, which means that the pressure of a single rendering will be concentrated on one rendering task. Complex 3D scenes will bring huge rendering pressure to the rendering host during ultra-high-definition rendering; when the rendering pressure exceeds the hardware specifications, such as the GPU video memory and running memory RAM are not enough to meet the rendering conditions, it will cause the rendering to fail to start or cause the software to crash during the rendering process; b) For 3D software that cannot render online and does not support VR cloud rendering, ultra-high-definition rendering will consume a lot of time and have low rendering efficiency; c) For some VR renderers, due to the consideration of the rendering pressure, there are certain resolution limitations, such as the rendering cannot exceed the standard VR8K resolution of 8192*4096, which cannot meet the needs of higher-definition VR rendering. Summary of the invention

[0003] The purpose of the present invention is to overcome the above problems existing in the prior art and greatly improve its technical effect on the basis of the original technology; the present invention provides a VR rendering method based on reconstructing the three-dimensional spatial relationship of the picture, the method comprising:

[0004] S100, according to rendering requirements, formulate camera array rules; the camera array rules refer to arranging multiple cameras according to a regular polyhedron, each camera is located at a vertex of the regular polyhedron, and a panoramic view covering 360 degrees is obtained through multiple cameras; specifically, according to the rendering requirements, the rendering array rules of the camera are formulated, and the formulated camera array enables the camera to render images in all camera perspectives under the same camera path, and under the same camera displacement, a panoramic view covering 360 degrees is achieved by deflecting the angles of multiple cameras; at the same time, the perspectives of adjacent cameras need to leave 10% overlapping parts of the screen for subsequent synthesis, and the same standard camera is rendered multiple times to achieve the effect of shooting a full panoramic angle; the same camera path refers to making the camera keep the formulated camera array unchanged and move along the same path; the deflection of the angles of multiple cameras refers to deflecting the angle of each camera so that the camera obtains image information of the corresponding viewing angle range, so that the multiple cameras cover a panoramic view of 360 degrees, and the viewing angle range is the corresponding viewing angle obtained by allocating the 360-degree panoramic view to each camera in the camera array; the same standard camera refers to the multiple cameras in the camera array having the same model and parameter settings.

[0005] S200. Setting up the three-dimensional software that needs to be rendered according to the established camera array rules; specifically, according to the camera array rules established in step S100, performing the same multiple camera array arrangement settings in the three-dimensional software to ensure that the number of cameras and the array arrangement rules are consistent with those established in the scene of step S100.

[0006] S300, executing a rendering task in a three-dimensional software to render the image in the camera; the rendering adopts a method of rendering multiple camera images in a camera array in batches; specifically, the three-dimensional software includes: Blender, Autodesk 3dsMax, Unreal Engine and Unity, etc.; the rendering adopts a method of rendering multiple camera images in a camera array in batches, including: rendering each camera image in the camera array separately, and improving the fusion effect by increasing the number of cameras in the array and the image overlap, and increasing the focal length value of a single camera to improve the picture clarity.

[0007] S400, reconstructing the three-dimensional spatial relationship of the image that has completed the rendering task in S300 in the synthesis software, and establishing pixel fusion between the images; the steps of reconstructing the three-dimensional spatial relationship in the synthesis software are: S401, establishing a center point; the center point refers to the center position of the camera array; S402, displacing the image according to the calculated moving pixels; the moving pixels refer to the pixels generated by the camera movement; S403, deflecting the image with the center point as the anchor point according to the deflection angle information in the camera array; the establishing pixel fusion between the images includes: using the image mask technology to perform fusion processing on each rendered image; specifically, the center point refers to the center position of the regular polyhedron composed of multiple cameras, and the moving pixels refer to the pixels generated by the camera movement; the synthesis software includes: Nuke, Adobe After Effects, Unreal Engine and Unity, etc.; the deflecting the image with the center point as the anchor point includes: before deflecting, calculating the displacement pixel amount of the camera image from the center point in the synthesis; after calculating the displacement pixel amount value, displacing the image with the center point as the anchor point; the calculation formula of the displacement pixel amount is:

[0008] Among them, β is the displacement pixel amount of the corresponding camera, α takes the larger value of the pixel height and width of the rendered image, tan is the tangent function, and FOV is the field of view.

[0009] Further, the fusion processing of each rendered image using the image mask technology includes: the image mask technology is a technology that controls the transparency and visibility of the image by masking a specific area of ​​the image; the image mask technology controls the display area and transparency of each image to achieve smooth transition and seamless image fusion; when the fusion processing of each rendered image is performed using the image mask technology, the expansion value and feathering value of the mask need to be calculated through the camera deflection angle and FOV in the camera array, and the calculation formula is:

[0010] Among them, a is the expansion value of the mask, which is a negative value. is the absolute value of a; b is the feathering value, which is a positive value, and both a and b are percentage values; tan is the tangent function, FOV is the camera field of view, and c is the camera deflection angle c.

[0011] After calculating the mask expansion value and feather value, set the mask value to the pixel ratio that needs to be gradually changed. , and make the absolute value of the mask expansion value the same as the feather value to achieve a gradual adjustment of 0-100% transparency in the fused pixels.

[0012] S500, executing a rendering task in the synthesis software; specifically, establishing a VR camera component of the synthesis software at the center point of the three-dimensional space established in S400, executing a rendering task on the fused image, and obtaining a final rendered product.

[0013] The beneficial effects of the present invention are:

[0014] The present invention provides a VR rendering method based on reconstructing three-dimensional spatial relationships of images; it has the following advantages:

[0015] 1. This method decomposes a single rendering pressure into multiple rendering tasks. At the same time, the number of decompositions can be changed according to demand. The more rendering tasks are decomposed, the higher the resolution can be achieved under the same rendering conditions, and the smaller the rendering pressure can be achieved under the same resolution, which solves the problem of serious hardware load for rendering ultra-high resolution under conventional GPU rendering process.

[0016] 2. Since the rendering tasks are decomposed, a single VR rendering task is decomposed into multiple ordinary rendering tasks, and it does not rely on rendering plug-ins. Because online rendering workflows or cloud rendering workflows can be used in this process, using this workflow can greatly reduce rendering time and improve production efficiency.

[0017] 3. Due to encoding limitations and rendering pressure limitations, the highest resolution of VR rendering in conventional VR rendering processes is often limited to 8192*4096, which cannot meet the needs of using a higher resolution. Since the rendering process of this method does not calculate the scene, it only performs a single simple calculation on the combination of sequences during the final rendering. Therefore, the resolution depends on the sequence resolution. The final rendering resolution is determined by the sequence resolution, breaking through the resolution limitation of the traditional workflow.

[0018] 4. This method adopts the mask fusion technology and provides a calculation method for obtaining the expansion value and feathering value of the mask through the camera deflection angle and FOV in the camera array. It can achieve a smooth and natural transition of the rendering boundary, solve the problem of rendering the special effects boundary of some VR renderers, and achieve a better finished product effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flow chart of a VR rendering method based on reconstructing three-dimensional spatial relationships of images.

[0020] Figure 2 A schematic diagram of the relationship between the camera FOV and the pixel information of the rendered material. DETAILED DESCRIPTION

[0021] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings; it should be understood that the specific embodiments given here are only used to illustrate and explain the present invention and cannot be used to limit the present invention.

[0022] It should be noted that many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention may also have other implementations and variations thereof. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0023] like Figure 1 As shown, a flow chart of a VR rendering method based on reconstructing a three-dimensional spatial relationship of an image according to an embodiment of the present invention; the flow chart includes: step S100, formulating a camera array rule according to rendering requirements; the camera array rule refers to arranging multiple cameras according to a regular polyhedron, each camera is located at a vertex of the regular polyhedron, and a panoramic view covering 360 degrees is obtained through multiple cameras; step S200, setting according to the formulated camera array rule in the three-dimensional software that needs to be rendered; step S300, executing a rendering task in the three-dimensional software to render the image in the camera; the rendering adopts the method of rendering multiple camera images in the camera array in batches; Step S400, reconstructing the three-dimensional spatial relationship of the image that has completed the rendering task in step S300 in the synthesis software, and establishing pixel fusion between images; the steps of reconstructing the three-dimensional spatial relationship in the synthesis software are: S401, establishing a center point; the center point refers to the center position of the camera array; S402, shifting the image according to the calculated moving pixels; the moving pixels refer to the pixels generated by the camera movement; S403, deflecting the image with the center point as the anchor point according to the deflection angle information in the camera array; the establishing pixel fusion between images includes: using the image mask technology to fuse each rendered image; step S500, executing the rendering task in the synthesis software.

[0024] In step S100, specifically, a rendering array rule of the camera is formulated according to the rendering requirements, and the formulated camera array enables the camera to render images in all camera perspectives under the same camera path, and under the same camera displacement, a panoramic perspective of 360 degrees is achieved by deflecting the angles of multiple cameras; at the same time, the perspectives of adjacent cameras need to leave 10% overlapping parts of the screen for subsequent synthesis, and the effect of capturing a complete panoramic angle is achieved by rendering the same standard camera multiple times; under the same camera path, it means that the camera keeps the formulated camera array unchanged and moves along the same path; by deflecting the angles of multiple cameras, it means that by deflecting the angle of each camera, the camera obtains image information of the corresponding viewing angle range, so that multiple cameras cover a panoramic perspective of 360 degrees, and the viewing angle range is the 360-degree panoramic perspective allocated to the corresponding viewing angle obtained by each camera in the camera array; for example, the camera array can be set to a regular octahedron, and the horizontal cameras in the camera array are set to No. 1, No. 2, No. 3 and No. 4. Camera No. 1, the four cameras in the horizontal direction are taken as a vertex of the regular octahedron, and the other two vertices of the regular octahedron are perpendicular to the horizontal direction, as cameras No. 5 and No. 6, and cameras No. 5 and No. 6 are perpendicular to the four cameras in the horizontal direction; it is stipulated that the horizontal camera is deflected horizontally, and the horizontal viewing angle range obtained by camera No. 1 is 0-90 degrees, the horizontal viewing angle range obtained by camera No. 2 is 90-180 degrees, the horizontal viewing angle range obtained by camera No. 3 is 180-270 degrees, and the horizontal viewing angle range obtained by camera No. 4 is 270-360 degrees; cameras No. 5 and No. 6 obtain the top and bottom viewing angles respectively; then by deflecting the angle of camera No. 1, image information within the viewing angle range of 0-90 degrees is obtained; the line connecting cameras No. 5 and No. 6 in the regular octahedron is taken as the deflection axis, and cameras No. 5 and No. 6 are deflected with the deflection axis as the center to obtain complete top and bottom viewing angles, so as to obtain a panoramic viewing angle covering 360 degrees through multiple cameras; the same standard camera refers to the multiple cameras in the camera array having the same models and parameter settings.

[0025] The array rule needs to include information such as the number of cameras in the camera array and the film back, focal length, FOV, aperture, and deflection angle of each camera.

[0026] In step S200, specifically, according to the camera array rule formulated in step S100, the same multiple camera array arrangement settings are performed in the three-dimensional software to ensure that the number of cameras and the array arrangement rules are consistent with those formulated in the step S100 scene; that is, the camera array rule formulated in step S100 is simulated in the three-dimensional software, so that the camera array arrangement simulated in the three-dimensional software and the camera array rule formulated in step S100 have the same number of cameras, and the same number of cameras have the same arrangement and the same viewing angle obtained by the corresponding cameras.

[0027] In step S300, specifically, the three-dimensional software includes: Blender, Autodesk 3ds Max, UnrealEngine and Unity, etc.; the rendering adopts the method of rendering multiple camera images in the camera array in batches, that is, rendering each camera image in the camera array separately, and improving the fusion effect by increasing the number of cameras in the array and the image overlap, and increasing the focal length value of a single camera to improve the picture clarity.

[0028] It should be pointed out that the traditional rendering method renders all cameras in the camera array at the same time. Although this method can improve the consistency of the rendered image and obtain a better fusion effect, it often brings huge rendering pressure to the rendering host when ultra-high-definition rendering is performed, the rendering efficiency is low, and it cannot meet the needs of higher-definition VR rendering. Therefore, the present invention adopts a rendering method that renders multiple camera images in the camera array in batches. This method can greatly disperse the rendering pressure and improve the fusion effect by increasing the number of cameras in the array and the image overlap. At the same time, it can increase the focal length value of a single camera, render a smaller image under the same resolution parameters, increase the sampling rate under a single rendering perspective, and improve the image clarity.

[0029] In step S400, specifically, when reconstructing the three-dimensional spatial relationship in the synthesis software, the center point refers to the center position of the regular polyhedron composed of multiple cameras, that is, the geometric center of the regular polyhedron; the moving pixel refers to the pixel generated by the camera movement; the synthesis software includes: Nuke, Adobe After Effects, Unreal Engine and Unity, etc.

[0030] Specifically, before the image is deflected with the center point as the anchor point, it is necessary to calculate the displacement pixel amount of the camera image from the center point in the synthesis; after calculating the displacement pixel amount value, the image is displaced with the center point as the anchor point; in step S100, as the camera in the camera array moves and deflects, the camera's FOV and deflection angle information are included, according to the camera's FOV and the pixel information of the rendered material, such as Figure 2 As shown, it is a schematic diagram of the relationship between the camera FOV and the pixel information of the rendering material of the present invention; the schematic diagram includes the camera FOV angle, that is, the field of view angle, the maximum pixel in the height or width, and the displacement distance; therefore, the calculation formula of the displacement pixel amount is:

[0031] Among them, β is the displacement pixel amount of the corresponding camera, α takes the larger value of the pixel height and width of the rendered image, tan is the tangent function, and FOV is the field of view angle.

[0032] After obtaining the displacement pixel amount of the corresponding camera, each rendered image is deflected with the center point as the anchor point according to the camera deflection angle established in step S100; each rendered image obtained in the rendering task is displaced and rotated in the three-dimensional space according to the above steps, that is, the reconstruction of the rendering task in the three-dimensional space is completed.

[0033] Further, in step S400, the image masking technology is a technology for controlling the transparency and visibility of an image by masking a specific area of ​​the image; the image masking technology controls the display area and transparency of each image to achieve smooth transition and seamless image fusion; when each rendered image is fused using the image masking technology, the expansion value and feathering value of the mask need to be calculated using the camera deflection angle and FOV in the camera array, and the calculation formula is:

[0034] Among them, a is the expansion value of the mask, which is a negative value. is the absolute value of a; b is the feathering value, which is a positive value, and both a and b are percentage values; tan is the tangent function, FOV is the camera field of view, and c is the camera deflection angle c.

[0035] It should be pointed out that the mask expansion value and feathering value contain the following principles: the camera's deflection angle is the actual effective information in 3D reconstruction, and the difference between FOV and the camera's deflection angle is used as the fused pixel information ratio of the picture; in the fused pixel ratio, it is necessary to adjust the transparency of the effective information pixels on the outer boundary of the fused pixel picture from 0 to 100% gradually, so the mask expansion and feathering functions in the mask function are introduced; set the mask value to the pixel ratio that needs to be gradually adjusted. , and at the same time make the absolute value of the mask expansion value the same as the feather value, and the mask expansion value is a negative number, that is, the mask is expanded inward, so that the gradual adjustment of 0-100% transparency in the fused pixels can be achieved; on this basis, the mask image mode of all images is adjusted to dissolve, and the boundary of each image and the effective pixels in the three-dimensional space reconstruction can be gradually dissolved to achieve a smooth and natural transition effect.

[0036] In step S500, a VR camera component of the synthesis software is established at the center point of the three-dimensional space established in S400, and a rendering task is performed on the fused image to obtain a final rendered product.

Claims

1. A VR rendering method based on reconstructing three-dimensional spatial relationships of images, characterized in that: The method comprises: S100, according to rendering requirements, formulate camera array rules; the camera array rules refer to arranging multiple cameras according to a regular polyhedron, each camera is located at a vertex of the regular polyhedron, and a panoramic view covering 360 degrees is obtained through multiple cameras; the camera array rules formulated according to rendering requirements include: formulating camera rendering array rules according to rendering requirements, the formulated camera array enables the camera to render images in all camera perspectives under the same camera path, and under the same camera displacement, by deflecting the angles of multiple cameras, a panoramic view covering 360 degrees is achieved; at the same time, the perspectives of adjacent cameras need to be synthesized for subsequent A 10% overlapped portion of the screen is reserved, and the same standard camera is rendered multiple times to achieve the effect of capturing a full panoramic angle; the same camera path means that the camera keeps the specified camera array unchanged and moves along the same path; the deflection of the angles of multiple cameras means that the camera obtains image information of the corresponding viewing angle range by deflecting the angle of each camera, so that multiple cameras cover a 360-degree panoramic viewing angle, and the viewing angle range is the corresponding viewing angle obtained by allocating the 360-degree panoramic viewing angle to each camera in the camera array; the same standard camera means that the models and parameter settings of multiple cameras in the camera array are all the same; S200, setting according to the established camera array rule in the 3D software that needs to be rendered; S300, executing a rendering task in a three-dimensional software to render an image in a camera; the rendering is performed by rendering multiple camera images in a camera array in batches; S400, reconstructing the three-dimensional spatial relationship of the image that has completed the rendering task in S300 in the synthesis software, and establishing pixel fusion between the images; the steps of reconstructing the three-dimensional spatial relationship in the synthesis software are: S401, establishing a center point; the center point refers to the center position of the camera array; S402, displacing the image according to the calculated moving pixels; the moving pixels refer to the pixels generated by the camera movement; S403, deflecting the image with the center point as the anchor point according to the deflection angle information in the camera array; the establishing pixel fusion between the images includes: using the image mask technology to fuse each rendered image; the reconstructing the three-dimensional spatial relationship in the synthesis software includes: the center point refers to the center position of the regular polyhedron composed of multiple cameras, and the moving pixels refer to the pixels generated by the camera movement; the synthesis software includes: Nuke, AdobeAfter Effects, Unreal Engine and Unity; the deflecting the image with the center point as the anchor point includes: before the deflection, calculating the displacement pixel amount of the camera image from the center point in the synthesis; after calculating the displacement pixel amount value, displacing the image with the center point as the anchor point; the calculation formula of the displacement pixel amount is: Among them, β is the displacement pixel amount of the corresponding camera, α takes the larger value of the pixel height and width of the rendered image, tan is the tangent function, and FOV is the field of view angle; The method of fusing each rendered image using the image mask technology includes: the image mask technology is a technology that controls the transparency and visibility of an image by masking a specific area of ​​the image; the image mask technology controls the display area and transparency of each image to achieve smooth transition and seamless image fusion; when fusing each rendered image using the image mask technology, the expansion value and feathering value of the mask need to be calculated through the camera deflection angle and FOV in the camera array, and the calculation formula is: Among them, a is the expansion value of the mask, which is a negative value. is the absolute value of a; b is the feathering value, which is a positive value, and both a and b are percentage values; tan is the tangent function, FOV is the camera field of view, and c is the camera deflection angle c; After calculating the mask expansion value and feather value, set the mask value to the pixel ratio that needs to be gradually changed. , and make the absolute value of the mask expansion value the same as the feather value to achieve a gradual adjustment of 0-100% transparency in the fused pixels; S500: Execute a rendering task in the synthesis software.

2. The VR rendering method based on reconstructing three-dimensional spatial relationships of images according to claim 1, characterized in that: The setting according to the established camera array rules in the three-dimensional software that needs to be rendered includes: according to the camera array rules established in step S100, the same multiple camera array arrangement settings are performed in the three-dimensional software to ensure that the number of cameras and the array arrangement rules are consistent with those established in the scene of step S100.

3. The VR rendering method based on reconstructing three-dimensional spatial relationships of images according to claim 1, characterized in that: The three-dimensional software includes: Blender, Autodesk 3ds Max, Unreal Engine and Unity; the rendering adopts the method of rendering multiple camera images in the camera array in batches, including: rendering each camera image in the camera array separately, improving the fusion effect by increasing the number of cameras in the array and the image overlap, and increasing the focal length value of a single camera to improve the picture clarity.

4. The VR rendering method based on reconstructing three-dimensional spatial relationships of images according to claim 1, characterized in that: The performing of the rendering task in the synthesis software includes: establishing a VR camera component of the synthesis software at the three-dimensional space center point established in S400, performing the rendering task on the fused image, and obtaining a final rendered product.

Images