Dense rendering method and device suitable for 3D light field display for 3D model of radiation field and computer equipment

Abstract

The invention discloses a dense rendering method and device suitable for 3D light field display for a 3D model of a radiation field and computer equipment, and relates to the field of 3D light field display, and the dense rendering method comprises the steps: selecting an image of a certain region in the 3D model as a central viewpoint image based on a preset observation angle; by taking the central viewpoint image as a reference and a viewpoint corresponding to the central viewpoint image as a central viewpoint, constructing an adjustment relationship between internal and external parameters of the virtual camera and a projection matrix, adjusting visual cone distribution according to different visual angles, and generating a plurality of simulation viewpoint images on the left side and the right side; the presented three-dimensional effect is three-dimensional vision which not only can meet the human eye motion parallax rule, but also conforms to the human binocular parallax characteristic. And the relative size relation of the depth values of scene points watched at different viewpoint positions can be changed, the correct space shielding relation is reflected, and the continuous perception of the human eye space structure is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of 3D light field display, and in particular to a method and device for dense rendering of a 3D model of a radiance field suitable for 3D light field display, and a computer device. BACKGROUND

[0002] In recent years, with the rapid development of deep learning and 3D Gaussian technology in the field of 3D reconstruction, the reconstruction accuracy and efficiency of 3D models have been significantly improved. However, how to present the stereoscopic visual effect of the reconstructed 3D model in an intuitive and efficient manner; at the same time, in the field of 3D light field display, the intermediate steps from the rendered 3D model to the display using a light field display have not been systematically and formally sorted out, and only tedious experiments can be used to continuously approach the correct offset, which brings many inconveniences and time loss to the experiment. The traditional method often has the problems of only being able to blindly experiment to fit, high computational complexity, or parallax effect not conforming to the human eye observation law when generating multi-viewpoint images, resulting in poor subsequent stereoscopic display effect.

[0003] Therefore, there is an urgent need for an efficient processing method for 3D models, which can generate multiple simulated viewpoint images conforming to the human eye parallax law based on a single center image. These viewpoint images need to meet the 3D encoding and ultimately adapt to the technical characteristics of the light field display to achieve high-quality naked-eye 3D display effect. SUMMARY

[0004] In view of the above problems in the prior art, the present application provides a method and device for dense rendering of a 3D model of a radiance field suitable for 3D light field display, and a computer device, which solves the problem of poor stereoscopic display effect of the 3D model of the radiance field in the prior art.

[0005] In order to achieve the above-mentioned purposes, the technical solution adopted by the present application is as follows: A method for dense rendering of a 3D model of a radiance field suitable for 3D light field display is provided, which comprises the following steps: Obtaining a 3D model of a radiance field, selecting a certain region image in the 3D model as a center viewpoint image based on a preset observation angle; Taking the center viewpoint image as a reference, taking the viewpoint corresponding to the center viewpoint image as a center viewpoint, constructing an adjustment relationship between the internal and external parameters of a virtual camera and a projection matrix, adjusting the view cone distribution according to different viewing angles, and generating multiple simulated viewpoint images on the left and right sides; 3D encoding the selected center viewpoint image and its corresponding simulated viewpoint images according to the viewpoint arrangement requirements, forming a final multi-viewpoint composite image, and completing the dense rendering.

[0006] Further, the specific method for constructing the adjustment relationship between the internal and external parameters of the virtual camera and the projection matrix is as follows: This is achieved by inversely transforming the offset of the virtual camera's intrinsic optical center and the translation of the virtual camera's extrinsic T matrix, and its expression is:

[0007]

[0008] in Indicates the location corresponding to the simulated viewpoint. The value after the change of the directional optical center; This indicates that the center viewpoint of the image is located at the center viewpoint. The original optical center value of the direction; express The simulated viewpoint of the location The change in the optical center of the direction, i.e., the intrinsic optical center of the virtual camera in The offset in direction; The translation matrix of the simulated viewpoint is in The value after the direction changes; The translation matrix representing the central viewpoint is in The original value of the direction represents the position of the virtual camera; express The simulated viewpoint translation matrix of the position is in The change in direction, i.e., the virtual camera extrinsic parameter T matrix in The amount of translation in the direction; Indicates the sequence number of the generated simulated viewpoint image; Direction is direction, direction or direction; These are the parameters for a 3D light field display.

[0009] Furthermore, when When the calculation formula takes the "+" sign, The calculation formula takes a "-" sign to fit the off-axis effect of the light field to the right of the central viewpoint; when When the calculation formula takes "-", The calculation formula is positive, and it is used to fit the off-axis effect of the light field to the left of the central viewpoint.

[0010] Furthermore, The simulated viewpoint of the location is Change in optical center in direction The angle of view at the furthest point, screen width, and virtual camera position. It is calculated as half of the angle that the direction can capture in the field of view.

[0011] Furthermore, The simulated viewpoint of the location is Change in optical center in direction The calculation expression is:

[0012] in For virtual cameras in Half of the angle that the direction can capture in the field of vision; It is the tangent function; The width of the screen; Number of viewpoints; This indicates the angle of view that can be observed from the farthest point of view. This is an offset angle adjustment factor used to adjust the offset viewpoint and the degree of stereoscopic effect; the virtual camera in... Half of the angle that the direction can capture in the field of vision Number of viewpoints Screen width Angle of view at the farthest point of view These are all parameters for 3D light field displays.

[0013] Furthermore, the virtual camera extrinsic parameter T matrix in Translation of direction The true depth distance of the target viewed by the virtual camera, and the virtual camera's position within the target. The focal length in the direction and the offset of the optical center of the simulated viewpoint are calculated.

[0014] Furthermore, the virtual camera extrinsic parameter T matrix in Translation of direction The calculation expression is:

[0015] in This represents the actual depth distance of the target being viewed by the virtual camera, and is a parameter of the 3D model. For virtual cameras in The focal length in the direction is a parameter of a 3D light field display. This is the zero-plane adjustment factor, used to adjust the zero plane and adapt to the offset angle adjustment factor. The changes.

[0016] Furthermore, when 3D encoding employs 3D Gaussian sputtering, the virtual camera's gaze distance to the target is the true depth distance. The value is obtained by estimating the average depth of the Gaussian point cloud in the 3D model of the radiation field.

[0017] An apparatus is provided for a dense rendering method suitable for 3D light field display based on a 3D model of a radiation field, comprising: The center view point image selection module is configured to acquire a 3D model of a radiation field, and select a certain area image in the 3D model as a center view point image based on a preset observation angle. The simulation view point image generation module is configured to take the center view point image as a reference, take a view point corresponding to the center view point image as a center view point, construct an adjustment relationship between internal and external parameters of a virtual camera and a projection matrix, adjust a view cone distribution according to different view angles, and generate a plurality of simulation view point images on the left and right sides. The multi-view point synthesis module is configured to perform 3D coding on the selected center view point image and the corresponding simulation view point images according to view point arrangement requirements, form a final multi-view point synthesis image, and complete dense rendering.

[0018] A computer device is provided, which includes a memory and a processor, and the memory stores a computer program, which, when executed by the processor, causes the processor to implement a dense rendering method for a 3D model of a radiation field suitable for 3D light field display.

[0019] The method can solve the problem that the existing radiation field model rendering cannot call an MVP matrix to implement off-axis rendering of a light field, take a center view point image as a reference, take a view point corresponding to the center view point image as a center view point, construct an adjustment relationship between internal and external parameters of a virtual camera and a projection matrix, adjust a view cone distribution according to different view angles, generate a plurality of simulation view point images on the left and right sides, and make the final multi-view point synthesis image match the 3D light field view point distribution law, so that correct 3D light field display is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a flowchart of the method; Figure 2 FIG. 2 is a schematic diagram of the adjustment relationship between internal and external parameters of a virtual camera and a projection matrix; Figure 3 FIG. 3 is an actual observation effect diagram of view point 0 (-50°) in the embodiment; Figure 4 FIG. 4 is an actual observation effect diagram of view point 48 (0°) in the embodiment; Figure 5 FIG. 5 is an actual observation effect diagram of view point 95 (50°) in the embodiment. DETAILED DESCRIPTION

[0021] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0022] As Figure 1 shown, the dense rendering method suitable for 3D light field display of the 3D model of the radiation field includes the following steps: S1, obtaining a 3D model of a radiation field, selecting a certain area image in the 3D model as a center view image based on a preset observation angle; S2, taking the center view image as a reference, taking the view point corresponding to the center view image as a center view point, constructing an adjustment relationship of virtual camera internal and external parameters and projection matrix, adjusting the view cone distribution according to different view angles, and generating a plurality of simulated view point images on the left and right sides; S3, 3D encoding the selected center view image and the corresponding simulated view point image according to the view point arrangement requirement, forming a final multi-viewpoint composite image, and completing the dense rendering.

[0023] The specific method for constructing the adjustment relationship of the virtual camera internal and external parameters and the projection matrix in step S2 is: The inverse change of the offset of the virtual camera internal parameter optical center and the translation of the virtual camera external parameter T matrix is realized, and the expression is:

[0024]

[0025] Wherein represents the value of the corresponding optical center after the change in the direction of the simulated view point; represents the original optical center value of the center view point of the center view image in the direction of represents the optical center change value of the simulated view point at the position in the direction of , that is, the offset of the virtual camera internal parameter optical center in the direction of represents the value of the translation matrix of the simulated view point after the change in the direction of ; represents the original value of the translation matrix of the center view point in the direction of , representing the position of the virtual camera; represents the value of the translation matrix of the simulated view point after the change in the direction of ; represents the original value of the translation matrix of the center view point in the direction of , representing the position of the virtual camera; represents the translation matrix of the simulated view point at the position in the direction of ; the change value of the direction, i.e. the virtual camera extrinsic parameter T matrix in the translation amount of the direction; the serial number of the generated simulated view point image; the direction is the direction, the direction or the direction; is the 3D light field display parameter.

[0026] The adjustment relationship between the above-mentioned virtual camera intrinsic and extrinsic parameters and the projection matrix combines depth information and perspective projection, and the parallax between adjacent view points conforms to the natural observation law, and the sequentially generated view point images have a stereoscopic form conforming to the human eye parallax.

[0027] In the embodiment, the light center offset amount and the translation amount of the virtual camera extrinsic parameter T matrix are inversely related in the direction. For example, assuming that the light center offset amount is positive and offset to the right, the virtual camera translation amount should be negative and the direction is to the left. The translation amount of the virtual camera is proportional to the offset amount of the light center, proportional to the depth of the virtual camera gaze point and the virtual camera, and inversely proportional to the focal length in the movement direction of the virtual camera. When the calculation formula of takes "+", the off-axis effect of the light field to the right of the center view point is fitted; when the calculation formula of takes "-", the off-axis effect of the light field to the left of the center view point is fitted.

[0028] In the embodiment, as shown in Figure 2 , taking the left as an example in the direction (the direction and the direction are the same), assuming that the intrinsic parameter of the center camera is , and the extrinsic parameter is R and T. Then the light center position of the center camera is . is the initially set virtual camera, and the italic is the virtual camera after changing the light center, then , , and the can be obtained by simultaneous solution. Similarly, the change amount of the light center of the N / 2th virtual camera is . For generating a new view point image to the right of the center view point, only is changed to , and the "+" in the calculation formula is changed to "-", and the "-" in the calculation formula is also changed to "+".

[0029] Based on the above derivation process, The simulated view point of the position is in The light center change value of the direction The half of the angle of view field that can be captured by the virtual camera in the direction of The corresponding calculation expression is:

[0030] Wherein is the half of the angle of view field that can be captured by the virtual camera in the direction of ; is the tangent function; is the screen width; is the number of view points; represents the angle of view field that can be observed under the farthest view point; is the offset angle adjustment factor, used for adjusting the offset view angle and the degree of stereoscopic display; the half of the angle of view field that can be captured by the virtual camera in the direction of ; , the number of view points , the screen width , the angle of view field that can be observed under the farthest view point are all the parameters of the 3D light field display.

[0031] The translation amount of the virtual camera extrinsic parameter T matrix in the direction of ; The real depth distance of the virtual camera gaze target, the focal length of the virtual camera in the direction of , and the offset amount of the simulated view point light center are calculated, and the calculation expression is:

[0032] Wherein is the real depth distance of the virtual camera gaze target, which belongs to the 3D model parameters; is the focal length of the virtual camera in the direction of , which belongs to the 3D light field display parameters; is the zero plane adjustment factor, used for adjusting the zero plane and adapting the change of the offset angle adjustment factor .

[0033] In the embodiment, when the reconstruction method adopts 3D Gaussian sputtering, the value of the real depth distance of the virtual camera gaze target is obtained by estimating the average depth of the Gaussian point cloud in the 3D model of the radiation field.

[0034] 3D coding can ensure that the view points on the light field display can be smoothly transitioned when the multi-view point image is continuously played, and the comfort degree of stereoscopic display is improved.

[0035] The embodiment also provides a device for dense rendering of a 3D model of a radiation field suitable for 3D light field display, which comprises: a central viewpoint image selection module configured to acquire the 3D model of the radiation field, and select a certain area image in the 3D model as a central viewpoint image based on a preset observation angle; a simulated viewpoint image generation module configured to take the central viewpoint image as a reference, take a viewpoint corresponding to the central viewpoint image as a central viewpoint, construct an adjustment relationship between virtual camera internal and external parameters and a projection matrix, adjust a view cone distribution according to different viewing angles, and generate a plurality of simulated viewpoint images on the left and right sides; a multi-viewpoint synthesis module configured to 3D encode the selected central viewpoint image and the corresponding simulated viewpoint images according to viewpoint arrangement requirements, form a final multi-viewpoint synthesis image, and complete the dense rendering.

[0036] The multi-viewpoint synthesis image is input into a 3D light field display, display adaptation and output are completed, and finally a naked-eye 3D effect can be presented.

[0037] The embodiment also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to enable the processor to implement the dense rendering method of the 3D model of the radiation field suitable for 3D light field display.

[0038] In the specific implementation process, multi-viewpoints are generated in the horizontal direction, the number of virtual cameras N = 96, the display width = 1600 (px), the included angle of the field of view that can be captured by one virtual camera (the visual angle of the display) = 0.01818 (rad), the opening angle of the field of view that can be viewed at the farthest viewpoint = 1.7453 (rad) = 100°, and the depth Z uses the line-of-sight depth Z_mean estimated by the 3D Gaussian point cloud center. On this basis, the simulated viewpoint images corresponding to the viewpoint 0, the viewpoint 48 and the viewpoint 95 are shown in Figure 3 、 Figure 4 and Figure 5 It can be seen that the application can efficiently generate multi-viewpoint images conforming to the parallax law of the human eye, the viewpoint images can meet the splicing requirements of the multi-tile image, and finally adapt to the technical characteristics of the light field display to achieve a high-quality naked-eye 3D display effect.

[0039] In summary, the application takes the center view point image as a reference, takes the view point corresponding to the center view point image as the center view point, constructs the adjustment relationship of the virtual camera internal and external parameters and the projection matrix, adjusts the view cone distribution according to different view angles, generates a plurality of simulation view point images on the left and right sides, and the stereoscopic effect presented is a kind of stereoscopic vision which can meet the human eye motion parallax law and conform to the human binocular parallax characteristics. The relative size relationship of the scene point depth values viewed at different view point positions can be changed, the correct spatial occlusion relationship can be reacted, and the continuous perception of the human eye spatial structure can be achieved.

Claims

1. A dense rendering method for a 3D model of a radiation field suitable for 3D light field display, characterized in that, Includes the following steps: Obtain a 3D model of the radiation field, and select an image of a certain region in the 3D model as the central viewpoint image based on a preset observation angle; Using the central viewpoint image as a reference and the viewpoint corresponding to the central viewpoint image as the central viewpoint, the adjustment relationship between the virtual camera's intrinsic and extrinsic parameters and the projection matrix is ​​constructed. The distribution of the view frustum is adjusted according to different viewing angles to generate multiple simulated viewpoint images on the left and right sides. The selected central viewpoint image and its corresponding simulated viewpoint image are 3D encoded according to the viewpoint arrangement requirements to form the final multi-viewpoint composite image, and then dense rendering is completed.

2. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 1, characterized in that, The specific method for constructing the relationship between the virtual camera's intrinsic and extrinsic parameters and the projection matrix is ​​as follows: This is achieved by inversely transforming the offset of the virtual camera's intrinsic optical center and the translation of the virtual camera's extrinsic T matrix, and its expression is: in Indicates the location corresponding to the simulated viewpoint. The value after the change of the directional optical center; This indicates that the center viewpoint of the image is located at the center viewpoint. The original optical center value of the direction; express The simulated viewpoint of the location is The change in the optical center of the direction, i.e., the intrinsic optical center of the virtual camera in The offset in direction; The translation matrix of the simulated viewpoint is shown in The value after the direction changes; The translation matrix representing the central viewpoint is in The original value of the direction represents the position of the virtual camera; express The simulated viewpoint translation matrix of the position is in The change in direction, i.e., the virtual camera extrinsic parameter T matrix in The amount of translation in the direction; Indicates the sequence number of the generated simulated viewpoint image; Direction is direction, direction or direction; These are the parameters for a 3D light field display.

3. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 2, characterized in that, when When the calculation formula takes "+", The calculation formula takes a "-" to fit the off-axis effect of the light field to the right of the central viewpoint; when When the calculation formula takes "-", The calculation formula is "+", used to fit the off-axis effect of the light field to the left of the central viewpoint.

4. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 2, characterized in that, The simulated viewpoint of the location is Change in optical center in direction The angle of view at the furthest point, screen width, and virtual camera position. It is calculated as half of the angle that the direction can capture in the field of view.

5. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 4, characterized in that, The simulated viewpoint of the location is Change in optical center in direction The calculation expression is: in For virtual cameras in Half of the angle that the direction can capture in the field of vision; It is the tangent function; The width of the screen; Number of viewpoints; This indicates the angle of view that can be observed from the farthest point of view. This is an offset angle adjustment factor used to adjust the offset viewpoint and the degree of stereoscopic effect; the virtual camera in... Half of the angle that the direction can capture in the field of vision Number of viewpoints Screen width Angle of view at the farthest point of view These are all parameters for 3D light field displays.

6. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 5, characterized in that, The virtual camera extrinsic parameter T matrix in Translation of direction The true depth distance of the target viewed by the virtual camera, and the virtual camera's position within the target. The focal length in the direction and the offset of the optical center of the simulated viewpoint are calculated.

7. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 6, characterized in that, The virtual camera extrinsic parameter T matrix in Translation of direction The calculation expression is: in This represents the actual depth distance of the target being viewed by the virtual camera, and is a parameter of the 3D model. For virtual cameras in The focal length in the direction is a parameter of a 3D light field display. This is the zero-plane adjustment factor, used to adjust the zero plane and adapt to the offset angle adjustment factor. The changes.

8. The dense rendering method for a 3D model of a radiation field suitable for 3D light field display according to claim 7, characterized in that, When 3D encoding uses 3D Gaussian sputtering, the virtual camera observes the true depth distance of the target. The value is obtained by estimating the average depth of the Gaussian point cloud in the 3D model of the radiation field.

9. An apparatus for performing dense rendering of a 3D model of a radiation field suitable for 3D light field display based on any one of claims 1 to 8, characterized in that, include: The central viewpoint image selection module is used to acquire a 3D model of the radiation field and select a region of the 3D model as the central viewpoint image based on a preset viewing angle. The simulated viewpoint image generation module is used to construct the adjustment relationship between the virtual camera's intrinsic and extrinsic parameters and the projection matrix, based on the central viewpoint image and the viewpoint corresponding to the central viewpoint image, and to adjust the view frustum distribution according to different viewing angles to generate multiple simulated viewpoint images on the left and right sides. The multi-viewpoint synthesis module is used to 3D encode the selected center viewpoint image and its corresponding simulated viewpoint image according to the viewpoint arrangement requirements to form the final multi-viewpoint synthesized image and complete the dense rendering.

10. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the dense rendering method for displaying a 3D model of a radiation field suitable for 3D light field, as described in any one of claims 1 to 8.

Images