A real-time rendering and stereoscopic display method and device suitable for 3D light field interaction

Abstract

The present invention discloses a real-time rendering and stereoscopic display method and device suitable for 3D light field interaction, and relates to the field of three-dimensional light field display technology. Use a brush module to perform spatial painting; capture the painting trajectory of the brush module through a spatial positioning module; use a real-time rendering module to map the spatial coordinates of the painting trajectory of the brush module to corresponding 3D light field pixel coding coordinates, and perform 3D light field pixel coding at the position corresponding to the obtained 3D light field pixel coding coordinates; the three-dimensional display module displays the three-dimensional image drawn by the brush module according to the 3D light field pixel coded coordinates and pixel information obtained by the real-time rendering module. The present invention directly maps the coordinates to the sub-pixels of the display according to the coding principle to achieve a real-time rendered three-dimensional stereoscopic display effect, without the need for dense viewpoint rendering, and the implementation method is simple and saves rendering time.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional light field display, and in particular to a real-time rendering and stereoscopic display method and device suitable for 3D light field interaction. Background Art

[0002] Human-computer interaction technology based on 2D displays has become increasingly mature. Users can freely realize functions such as painting and model control through touch, electronic pen, mouse and other interactive methods. However, in the field of three-dimensional display, it is difficult to realize real-time interaction of 3D light field due to the high complexity of three-dimensional light field data rendering and high computing power consumption. At present, CN201811329328.2 proposes a three-dimensional painting method and device. The method of this patent is only applicable to two-dimensional display, but cannot be applied to three-dimensional display, and only adjusts the three-dimensional height of each pixel on the path according to the height transformation and the scale, and cannot contain more information of the light field and provide multi-viewpoint viewing capabilities. The software implementation itself cannot provide a more real-time rendering method. CN202111320433.1 proposes a "three-dimensional modeling software framework and design method based on gesture interaction". This method improves the 2D rendering and interaction rate of 3D models by integrating dynamic link libraries, but it is difficult to apply to 3D light field display rendering of models or painting trajectories.

[0003] The traditional 3D light field data rendering process is divided into three steps: determining the viewpoint position, rendering dense viewpoint images, and three-dimensional light field encoding. Among them, rendering dense viewpoint images takes the longest time and has the highest computational complexity, and there is a large amount of rendering redundancy. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a real-time rendering and stereoscopic display method and device suitable for 3D light field interaction, which solves the problems of the prior art such as long rendering time, high calculation complexity and difficulty in realizing real-time 3D light field interaction.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a real-time rendering and stereoscopic display method suitable for 3D light field interaction, comprising the following steps:

[0006] S1. Use the brush module to paint in space;

[0007] S2, using the spatial positioning module to capture the painting trajectory of the brush module;

[0008] S3, using a real-time rendering module to establish a mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel coding coordinates to obtain the 3D light field pixel coding coordinates;

[0009] S4, performing 3D light field pixel encoding at a position corresponding to the obtained 3D light field pixel encoding coordinates, that is, extracting a specific sub-pixel in the disparity map sequence to be displayed;

[0010] S5. Arrange the obtained specific sub-pixels in a corresponding pattern according to the parameters of the display in the three-dimensional display module to generate a new 3D light field synthesis image and display it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory.

[0011] Furthermore, the mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel encoding coordinates is:

[0012]

[0013]

[0014] Where P is the grating pitch and the grating inclination is ; fmod is the modulo function; the spatial position coordinates of the captured brush module are (X, Y, Z), and the 3D light field pixel encoding coordinates corresponding to the spatial position coordinates of the brush module are , represents the kth sub-pixel in the uth column and the vth row pixel; , is the focal length of the spatial positioning module in the x and y axis directions, width is the width of the display in the three-dimensional display module, It is the angle of the visual field at the farthest viewing point; is the visual angle that can be captured by the spatial positioning module; N is the number of viewpoints, n is the viewpoint number; cy is the optical center in the y-axis direction; Represents the poses of different viewpoint numbers in the spatial positioning module; They represent the positions of the spatial positioning module in the Y and Z directions respectively; △ is the distance between viewpoints in the X direction.

[0015] The present invention also provides a device capable of implementing the above method, comprising:

[0016] Brush module, used for space painting;

[0017] The spatial positioning module is used to capture the painting trajectory of the brush module;

[0018] A real-time rendering module is used to map the spatial coordinates of the painting trajectory of the brush module into corresponding 3D light field pixel coding coordinates, and perform 3D light field pixel coding at the position corresponding to the obtained 3D light field pixel coding coordinates, that is, to extract specific sub-pixels in the disparity map sequence to be displayed;

[0019] The three-dimensional display module is used to arrange the specific sub-pixels obtained in a corresponding pattern according to the parameters of the display in the three-dimensional display module to generate a new 3D light field synthesis image and display it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory.

[0020] Furthermore: the brush module realizes aerial painting by freely changing its position in the three-dimensional space within the monitoring range of the spatial positioning module, and the feature captured by the spatial positioning module is the spatial position of the brush module; the functions of the brush module include changing the brush color, changing the brush material, changing the brush thickness, selecting and moving 3D images.

[0021] Furthermore: the spatial positioning module includes multiple depth cameras or binocular cameras arranged around the display in the three-dimensional display module; the spatial positioning module uses the center of the display screen in the three-dimensional display module as the origin, the horizontal direction as the x coordinate, the vertical direction as the y coordinate, and the vertical direction as the z coordinate for position calibration.

[0022] Furthermore: the three-dimensional display module includes a three-dimensional display, and its types include a three-dimensional light field display, a free stereoscopic display, and an integrated imaging display.

[0023] Furthermore, the expression for mapping the spatial coordinates of the painting trajectory of the brush module to the corresponding 3D light field pixel encoding coordinates is:

[0024]

[0025]

[0026] Where P is the grating pitch and the grating inclination is ; fmod is the modulo function; the spatial position coordinates of the captured brush module are (X, Y, Z), and the 3D light field pixel encoding coordinates corresponding to the spatial position coordinates of the brush module are , represents the kth sub-pixel in the uth column and the vth row pixel; , is the focal length of the spatial positioning module in the x and y axis directions, width is the width of the display in the three-dimensional display module, It is the angle of the visual field at the farthest viewing point; is the visual angle that can be captured by the spatial positioning module; N is the number of viewpoints, n is the viewpoint number; cy is the optical center in the y-axis direction; Represents the poses of different viewpoint numbers in the spatial positioning module; They represent the positions of the spatial positioning module in the Y and Z directions respectively; △ is the distance between viewpoints in the X direction.

[0027] The beneficial effects of the present invention are as follows: the present invention innovatively proposes a method suitable for real-time rendering and stereoscopic display of 3D light field painting, and uses a movement trajectory calculation method different from the prior art to perform three-dimensional painting trajectory mapping, uses a 3D multi-viewpoint algorithm to achieve better restoration of three-dimensional information, and directly maps coordinates to sub-pixels of the display according to the coding principle to achieve a real-time rendering display effect, without the need for dense viewpoint rendering, saving rendering time. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of the method proposed by the present invention;

[0029] Figure 2 The interaction and rendering frame rates of the method proposed by the present invention when the number of rendering viewpoints is 25;

[0030] Figure 3 The interactive,rendering frame rate of the traditional method when the number of rendering,viewpoints is 25;

[0031] Figure 4 The frame rate of interaction and rendering of the method proposed by the present invention when the number of rendering viewpoints is 50;

[0032] Figure 5 The interactive,rendering frame rate of the traditional method when the number of rendering,viewpoints is 50;

[0033] Figure 6 The interaction and rendering frame rates of the method proposed in the present invention when the number of rendering viewpoints is 100;

[0034] Figure 7 It is the interactive,rendering frame rate of the traditional method when the number of rendering,viewpoints is 100. DETAILED DESCRIPTION

[0035] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0036] like Figure 1 As shown, the present invention provides a real-time rendering and stereoscopic display method suitable for 3D light field interaction, comprising the following steps:

[0037] S1. Use a brush module to perform spatial painting; the brush module includes but is not limited to a top-emitting laser pen, an electronic pen, special gestures, etc.

[0038] S2, using the spatial positioning module to capture the painting trajectory of the brush module;

[0039] S3, using a real-time rendering module to establish a mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel coding coordinates, to obtain the 3D light field pixel coding coordinates; the mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel coding coordinates is:

[0040]

[0041]

[0042] Where P is the grating pitch and the grating inclination is ; fmod is the modulo function; the spatial position coordinates of the captured brush module are (X, Y, Z), and the 3D light field pixel encoding coordinates corresponding to the spatial position coordinates of the brush module are , represents the kth sub-pixel in the uth column and the vth row pixel; , is the focal length of the spatial positioning module in the x and y axis directions, width is the width of the display in the three-dimensional display module, It is the angle of the visual field at the farthest viewing point; is the field of view angle that can be captured by the camera in the spatial positioning module; N is the number of viewpoints, n represents the viewpoint number; cy represents the optical center in the y-axis direction; Respectively represent the position of the spatial positioning module in the Y and Z directions; △ is the distance between viewpoints in the X direction; Represents the poses of different viewpoint numbers in the spatial positioning module.

[0043] S4, performing 3D light field pixel encoding at a position corresponding to the obtained 3D light field pixel encoding coordinates, that is, extracting a specific sub-pixel in the disparity map sequence to be displayed;

[0044] S5. Arrange the obtained specific sub-pixels in a corresponding pattern according to the parameters of the display in the three-dimensional display module to generate a new 3D light field synthesis image and display it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory.

[0045] The present invention also provides a device suitable for the above method, comprising:

[0046] Brush module, used for space painting;

[0047] The spatial positioning module is used to capture the painting trajectory of the brush module; the brush module can freely change its position in the three-dimensional space within the monitoring range of the spatial positioning module to achieve aerial painting, and the features captured by the spatial positioning module are the spatial position of the brush module; the functions of the brush module include changing the brush color, changing the brush material, changing the brush thickness, selecting and moving 3D images. The spatial positioning module has a depth detection function, including multiple depth cameras or binocular cameras placed around the display in the three-dimensional display module; the spatial positioning module uses the center of the display screen in the three-dimensional display module as the origin, the horizontal direction as the x coordinate, the vertical direction as the y coordinate, and the vertical direction as the z coordinate for position calibration.

[0048] A real-time rendering module is used to map the spatial coordinates of the painting trajectory of the brush module into corresponding 3D light field pixel coding coordinates, and perform 3D light field pixel coding at the position corresponding to the obtained 3D light field pixel coding coordinates, that is, to extract specific sub-pixels in the disparity map sequence to be displayed;

[0049] The three-dimensional display module is used to generate a new 3D light field synthesis image by arranging the obtained specific sub-pixels in a corresponding pattern according to the parameters of the display in the three-dimensional display module and displaying it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory. The three-dimensional display module includes a three-dimensional display, and its types include a three-dimensional light field display, a free stereoscopic display, and an integrated imaging display.

[0050] In one embodiment of the present invention, an electronic pen with a light-emitting tip is used as a paintbrush, the resolution of the display in the 3D display module used is 7680*4320, the single-view resolution is 1920*1080, the screen size is 65 inches, the number of lines per inch is 15.086, and the grating inclination angle is is 0.124. At the same time, the focal lengths in the x and y axes are set to , the optical center in the y-axis direction , the viewing angle of the farthest viewpoint , the field of view angle that can be captured by the camera in the spatial positioning module , the interval between viewpoints The given parameters are optional and can be adjusted according to actual conditions.

[0051] Under the above parameter settings, according to the test, when using the brush module to interactively paint the model, Figure 2 , Figure 3 As shown, if the number of rendering viewpoints , the interaction and rendering frame rates of the method proposed in the present invention and the traditional method are both about 36fps, and the rendering frame rate ratio (percentage) of the method proposed in the present invention and the traditional method is 102.4%; Figure 4 , Figure 5 As shown, when rendering the number of viewpoints The interactive and rendering frame rate of the method proposed in the present invention is 36fps, while the rendering frame rate of the traditional method is only 16.1fps. The ratio (percentage) of the rendering frame rate of the method proposed in the present invention to that of the traditional method is 223.6%. Figure 6 , Figure 7 As shown, when rendering the number of viewpoints When the frame rate of interaction and rendering of the method proposed in the present invention is 36.1fps, while the frame rate of rendering of the traditional method is only 5.9fps, and the ratio (percentage) of the rendering frame rate of the method proposed in the present invention to the traditional method is 611.86%. This shows that the real-time rendering method for 3D light field interaction proposed in the present invention is superior to the traditional 3D light field rendering method, and is less affected by the number of viewpoints, and can realize real-time interaction and display of 3D light fields (especially 3D light fields with large viewing angles and dense viewpoints).

[0052] In summary, the present invention has low computational complexity and directly maps the painting trajectory to the sub-pixels of the display to achieve real-time rendering effects, which is more real-time and does not require intensive viewpoint rendering. It not only saves rendering time, but also provides multi-viewpoint viewing capabilities and improves the efficiency of 3D display.

Claims

1. A real-time rendering and stereoscopic display method suitable for 3D light field interaction, characterized in that: The following steps are involved: S1. Use the brush module to paint in space; S2, using the spatial positioning module to capture the painting trajectory of the brush module; S3, using a real-time rendering module to establish a mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel coding coordinates to obtain the 3D light field pixel coding coordinates; S4, performing 3D light field pixel encoding at a position corresponding to the obtained 3D light field pixel encoding coordinates, that is, extracting a specific sub-pixel in the disparity map sequence to be displayed; S5, according to the parameters of the display in the three-dimensional display module, the obtained specific sub-pixels are arranged in a corresponding regularity to generate a new 3D light field synthesis image and display it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory; The mapping relationship between the spatial coordinates of the painting trajectory and the 3D light field pixel encoding coordinates is: Where P is the grating pitch and the grating inclination is ; fmod is the modulo function; the spatial position coordinates of the captured brush module are (X, Y, Z), and the 3D light field pixel encoding coordinates corresponding to the spatial position coordinates of the brush module are , represents the kth sub-pixel in the uth column and the vth row pixel; , is the focal length of the spatial positioning module in the x and y axis directions, width is the width of the display in the three-dimensional display module, It is the angle of the visual field at the farthest viewing point; is the visual angle that can be captured by the spatial positioning module; N is the number of viewpoints, n represents the viewpoint number; cy represents the optical center in the y-axis direction; Respectively represent the position of the spatial positioning module in the Y and Z directions; △ is the distance between viewpoints in the X direction; Represents the poses of different viewpoint numbers in the spatial positioning module.

2. A real-time rendering and stereoscopic display device suitable for 3D light field interaction, characterized in that: include: Brush module, used for space painting; The spatial positioning module is used to capture the painting trajectory of the brush module; A real-time rendering module is used to map the spatial coordinates of the painting trajectory of the brush module into corresponding 3D light field pixel coding coordinates, and perform 3D light field pixel coding at the position corresponding to the obtained 3D light field pixel coding coordinates, that is, to extract specific sub-pixels in the disparity map sequence to be displayed; A three-dimensional display module, which is used to generate a new 3D light field synthesis image by arranging the obtained specific sub-pixels in a corresponding pattern according to the parameters of the display in the three-dimensional display module and displaying it on the three-dimensional display module, thereby realizing real-time rendering and three-dimensional stereoscopic display of the painting trajectory; The expression for mapping the spatial coordinates of the painting trajectory of the brush module to the corresponding 3D light field pixel encoding coordinates is: Where P is the grating pitch and the grating inclination is ; fmod is the modulo function; the spatial position coordinates of the captured brush module are (X, Y, Z), and the 3D light field pixel encoding coordinates corresponding to the spatial position coordinates of the brush module are , represents the kth sub-pixel in the uth column and the vth row pixel; , is the focal length of the spatial positioning module in the x and y axis directions, width is the width of the display in the three-dimensional display module, It is the angle of the visual field at the farthest viewing point; is the visual angle that can be captured by the spatial positioning module; N is the number of viewpoints, n represents the viewpoint number; cy represents the optical center in the y-axis direction; Respectively represent the position of the spatial positioning module in the Y and Z directions; △ is the distance between viewpoints in the X direction; Represents the poses of different viewpoint numbers in the spatial positioning module.

3. The real-time rendering and stereoscopic display device suitable for 3D light field interaction according to claim 2, characterized in that: The brush module can paint in the air by freely changing its position in the three-dimensional space within the monitoring range of the spatial positioning module. The features captured by the spatial positioning module are the spatial position of the brush module. The functions of the brush module include changing the brush color, changing the brush material, changing the brush thickness, selecting and moving 3D images.

4. The real-time rendering and stereoscopic display device suitable for 3D light field interaction according to claim 2, characterized in that: The spatial positioning module includes multiple depth cameras or binocular cameras arranged around the display in the three-dimensional display module; the spatial positioning module uses the center of the display screen in the three-dimensional display module as the origin, the horizontal direction as the x coordinate, the vertical direction as the y coordinate, and the vertical direction as the z coordinate for position calibration.

5. The real-time rendering and stereoscopic display device suitable for 3D light field interaction according to claim 2, characterized in that: The three-dimensional display module includes a three-dimensional display, and its types include a three-dimensional light field display, a free stereoscopic display, and an integrated imaging display.

Images