A three-dimensional scene display method, system, device and medium

Abstract

This application discloses a three-dimensional scene display method, system, device, and medium, relating to the display field. The method includes: stratifying the three-dimensional scene according to the distance between each area in the three-dimensional scene and the zero plane, and reducing the redundancy of content resources in the distant layers; performing sparse viewpoint acquisition on each scene layer based on the distance between each scene layer and the zero plane to obtain a sparse low-resolution viewpoint image of each scene layer; performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer to obtain a dense high-resolution viewpoint image of each scene layer; and encoding and synthesizing the dense high-resolution viewpoint image according to the display parameters of the light field display and the front-to-back positional relationship between multiple scene layers to obtain a dense viewpoint high-resolution light field encoding map, which is displayed on the light field display. This application reduces the consumption of data volume and computing resources, and can generate high-quality three-dimensional visual effects in large-scale scenes in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of display, in particular to a three-dimensional scene display method, system, device and medium. BACKGROUND

[0002] A light field image is usually composed of tens to tens of thousands of viewpoint images. This highly complex structure means that when rendering a large-scale scene light field, the amount of computation required is extremely large and complex, making real-time generation very difficult. Each viewpoint image not only needs to be processed independently, but also needs to consider the relationship and subtle changes between viewpoints, which further increases the computational burden. In addition, factors such as lighting, shadows and object movement in a large-scale scene also have a significant impact on rendering quality, resulting in decreased processing efficiency. Therefore, how to reduce the computational complexity while maintaining high image quality has become a major challenge in current light field rendering research. SUMMARY

[0003] The purpose of the present application is to provide a three-dimensional scene display method, system, device and medium, which can reduce the consumption of data volume and computing resources, and generate high-quality three-dimensional visual effects in real time in a large-scale scene.

[0004] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a three-dimensional scene display method, comprising:

[0006] Layering the three-dimensional scene according to the distance of each region in the three-dimensional scene to the zero plane, and performing content resource redundancy reduction on the long-distance layer to obtain a plurality of scene layers; the long-distance layer is a scene layer with a distance to the zero plane greater than a set threshold;

[0007] Based on the distance of each scene layer to the zero plane, sparse viewpoint collection is performed on each scene layer respectively to obtain a sparse low-resolution viewpoint image of each scene layer;

[0008] Viewpoint super-resolution processing is performed on the sparse low-resolution viewpoint image of each scene layer respectively to obtain a dense high-resolution viewpoint image of each scene layer; the resolution of the sparse low-resolution viewpoint image is less than the resolution of the dense high-resolution viewpoint image, and the number of viewpoints in the sparse low-resolution viewpoint image is less than the number of viewpoints in the dense high-resolution viewpoint image;

[0009] According to the display parameters of the light field display and the front and rear position relationship between the plurality of scene layers, the dense high-resolution viewpoint images of the plurality of scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field encoding image;

[0010] The dense viewpoint high-resolution light field encoding image is displayed on the light field display.

[0011] In a second aspect, the present application provides a three-dimensional scene display system, comprising:

[0012] a scene layering module configured to layer the three-dimensional scene according to distances of regions in the three-dimensional scene to a zero plane, and to reduce redundancy of content resources of a far distance layer to obtain a plurality of scene layers; the far distance layer is a scene layer with a distance to the zero plane greater than a set threshold;

[0013] a sparse viewpoint collection module configured to collect sparse low-resolution viewpoint images of each scene layer based on distances of the scene layers to the zero plane;

[0014] a viewpoint super-resolution module configured to perform viewpoint super-resolution processing on the sparse low-resolution viewpoint images of each scene layer to obtain dense high-resolution viewpoint images of each scene layer;

[0015] an encoding and synthesizing module configured to encode and synthesize the dense high-resolution viewpoint images of the plurality of scene layers according to display parameters of a light field display and a front-back positional relationship between the plurality of scene layers to obtain a dense viewpoint high-resolution light field encoding image;

[0016] a display module configured to display the dense viewpoint high-resolution light field encoding image on the light field display.

[0017] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene display method described above.

[0018] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the three-dimensional scene display method described above.

[0019] According to the embodiments provided in the present application, the present application has the following technical effects:

[0020] The present application provides a three-dimensional scene display method, system, device and medium, the three-dimensional scene is layered according to distances of regions in the three-dimensional scene to a zero plane, and the redundancy of content resources of a far distance layer is reduced, thereby reducing redundant data in the rendering process, sparse viewpoint collection is performed on each scene layer based on distances of the scene layers to the zero plane, thereby reducing data volume and consumption of computing resources, and viewpoint super-resolution processing is performed on sparse low-resolution viewpoint images, thereby achieving real-time display effect with clear details and accurate parallax in multi-view dynamic display. Therefore, the present application can generate high-quality three-dimensional visual effect in a large-scale scene in real time. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0022] Figure 1 An application environment diagram of a three-dimensional scene display method in an embodiment of the present application;

[0023] Figure 2 An overall flowchart of a three-dimensional scene display method provided in an embodiment of the present application;

[0024] Figure 3 A detailed flowchart of a three-dimensional scene display method provided in an embodiment of the present application;

[0025] Figure 4 A generation process diagram of a dense viewpoint high-resolution light field coding diagram in an embodiment of the present application;

[0026] Figure 5 An illustration of mesh slicing grouping in an embodiment of the present application;

[0027] Figure 6 A functional module diagram of a three-dimensional scene display system provided in an embodiment of the present application;

[0028] Figure 7 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0030] In order to solve the problem of performance degradation in adaptive naked-eye 3D light field display, it is urgent to deeply study the layered variable resolution sampling technology. The layered variable resolution sampling technology aims to reduce redundant viewpoint image information by dynamically adjusting the sampling density of different viewpoints, thereby significantly optimizing the rendering efficiency.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0032] The three-dimensional scene display method provided in the embodiments of the present application can be applied to the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be separately arranged, integrated on the server 104, placed on a cloud or other servers. The terminal 102 can send a three-dimensional scene to be displayed to the server 104. After receiving the three-dimensional scene to be displayed, the server 104 generates a dense viewpoint high-resolution light field encoding image and displays it on a light field display. In addition, in some embodiments, the three-dimensional scene display method can also be implemented by the server 104 or the terminal 102 alone.

[0033] The terminal 102 can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0034] In an exemplary embodiment, as shown in Figures 2 to 4 A three-dimensional scene display method is provided, which is executed by a computer device, specifically by a terminal or a server, or by both the terminal and the server. In the embodiments of the present application, the method is applied to the server 104 in Figure 1 The following steps 201 to 205 are included.

[0035] Step 201: Layering the three-dimensional scene according to the distance of each region in the three-dimensional scene to the zero plane, and reducing the redundancy of the content resources of the far distance layer to obtain a plurality of scene layers. The far distance layer is a scene layer with a distance greater than a set threshold to the zero plane.

[0036] The in-and-out screen distance of each part of the scene when the 3D scene is displayed on the light field display is determined. The 3D scene is layered according to the in-and-out screen distance of the scene and the display depth characteristics of the current light field display, and the scene layer far from the zero plane is reduced in resource redundancy.

[0037] The method of layering and the number of scene layers can be flexibly determined according to the display effect and display characteristics of the light field display. In 3D light field display, due to the inherent characteristics of light field display, the area closest to the zero plane has the best visual effect. The larger the out-of-screen / in-screen distance, i.e. the greater the distance from the zero plane, the more blurred the visual information and the worse the visual effect. Different three-dimensional stereoscopic displays have different display depth ranges, so before layering the scene according to the visual information, the display depth range of the display should be measured, as well as the visual effect of the display area at different depth of field. The judgment of visual effect can be quantified using objective image quality evaluation indexes including but not limited to peak signal-to-noise ratio (PSNR), structural similarity (SSIM), etc. The evaluated index value can also be used as the basis for scene layering according to the subjective feeling of the viewer.

[0038] In an exemplary embodiment, according to the display depth range of the three-dimensional light field display and the visual display effect at each display depth, the scene is finely layered and divided into multiple levels, each level corresponding to a different area in the scene. The importance of the scene is determined by its distance from the zero plane. The closer to the zero plane, the higher the importance and the finer the display of the scene, and the better the visual effect. The out-of-screen / in-screen area near the zero plane and the zero plane, which has small crosstalk and high visual display quality, is a group. The out-of-screen / in-screen scene area far from the zero plane, which has slightly poor visual display quality, is a group. The out-of-screen / in-screen scene area far from the zero plane, which has very poor visual display effect, is a group.

[0039] After layering the scene, the resource optimization is performed on each scene layer, especially on the scene layer far from the zero plane.

[0040] The 3D scene content resource is composed of basic primitive grids, materials and environmental lighting information. In order to improve the display performance of 3D light field, the grid structure and center camera parameters of 3D virtual scene are input, the Levels of Detail (LOD) structure is constructed through grid preprocessing, and the content resource is optimized. Specifically, the basic principle of light field content resource optimization is to allocate more grid faces to the part with strong 3D light field display performance, and allocate fewer grid faces to the part with weak performance. This can effectively reduce information redundancy and reduce the computational pressure during content generation.

[0041] The LOD and adaptive mesh slicing technology is adopted to preprocess the high-precision scene model into adaptive meshes, and the mesh density is adjusted in real time according to the distance between the scene and the zero plane, so as to optimize the rendering effect. The scene mesh density is high when the distance to the zero plane is short, and the display precision is high. The scene mesh density is low when the distance to the zero plane is far, and the display precision is low. Therefore, according to the display characteristics of the light field display, the layered scene is optimized at multiple levels, the rendering consumption of the scene far from the zero plane layer is reduced, and the computing resources are saved.

[0042] Since the high-precision image often contains redundant information, the adaptive mesh slicing technology can effectively merge these redundant meshes and reduce the amount of calculation consumed during camera acquisition (rendering). As shown in Figure 5 The mesh slicing divides the scene model into multiple segments (usually 64x64) with the same number of triangles, which helps to balance details and performance, so that the scene can be quickly adjusted at different viewing distances and angles. Figure 5 The left side is a schematic diagram of mesh slicing, and the right side is a schematic diagram of grouping 18 segments into a segment group.

[0043] Step 202, based on the distance between each scene layer and the zero plane, sparse view acquisition is performed on each scene layer respectively, and sparse low-resolution view images of each scene layer are obtained.

[0044] In an exemplary embodiment, step 202 includes steps 301 to 303.

[0045] Step 301, for any scene layer, according to the distance between the scene layer and the zero plane, the number of cameras and the resolution of the scene layer are determined. The closer the distance to the zero plane, the more the number of cameras and the higher the resolution of the scene layer.

[0046] According to the distance between each scene layer and the zero plane, the number of cameras and the resolution of each scene layer are flexibly set to ensure the best visual effect and computing efficiency. When processing complex scenes, good performance can still be maintained, and delay or lag phenomenon caused by excessive resource consumption can be avoided. Through flexible configuration strategy, not only the efficiency of information acquisition is significantly improved, but also the information of important areas can be fully captured, and the resource consumption is effectively reduced, so that higher response speed and excellent image quality can be realized in real-time application. The advantage of layered design is also reflected in its adaptability. With the development of technology and changes in application requirements, the camera configuration and resolution setting of each layer can be dynamically adjusted according to the specific scene characteristics and user requirements.

[0047] The scene layer close to the zero plane has high display quality on the light field display with little crosstalk effect. Therefore, a large number of cameras with high resolution are selected to capture rich details and dynamic information, so that the audience can feel the reality and delicacy of the picture when watching, which not only helps to show the texture and light and shadow changes of the object, but also enhances the audience's immersive experience, making the interaction more smooth and natural.

[0048] The scene layer far from the zero plane has a large out-of-screen distance and large crosstalk effect, resulting in poor visual effect and relatively low demand for details. Therefore, a small number of cameras with low resolution can be selected to effectively reduce the computational burden and optimize resource utilization, saving resources consumed in rendering.

[0049] In step 302, the viewpoint position of the scene layer is determined according to the content resource of the scene layer.

[0050] In an exemplary embodiment, more dense cameras are placed in complex scene positions for collection, and more sparse cameras are placed in simple places, so that a smaller number of cameras are arranged at the viewpoint position that can capture more scene information, and the redundancy of collected information and the consumption of collection time are minimized.

[0051] The application uses a sparse collection strategy based on a viewpoint measurement method to efficiently collect scene information by analyzing the geometric features of a three-dimensional scene model and calculating the importance of the vertices, and selecting appropriate viewpoint positions. Since the three-dimensional scene model is generally expressed by a fitted geometric patch, and the geometric content is also an important aspect of the perception model, the viewpoint measurement method measures the amount of information that can be observed at different positions by observing and detecting the geometric features of the three-dimensional scene, with the purpose of selecting a viewpoint that contains more information, thereby achieving efficient collection of scene information.

[0052] The sparse collection strategy based on the viewpoint measurement method aims to find the viewpoint position that can collect the most scene information, so as to achieve more scene information collection with fewer viewpoints. Specifically, the viewpoint entropy method can be used to calculate the Gaussian weighted curvature of each vertex in the three-dimensional scene model at multiple scales, and the importance of the vertex is measured based on this. The sum of the importance of the visible vertices of each viewpoint is used to measure the quality of the viewpoint, so as to select appropriate sparse collection positions. Other viewpoint measurement methods can also be selected to realize the selection of sparse collection positions.

[0053] In step 303, based on the number of cameras, the resolution and the viewpoint position of the scene layer, a sparse low-resolution viewpoint image of the scene layer is collected.

[0054] Specifically, a sparse camera arrangement is used to collect scene layers at low resolution to obtain sparse low-resolution view images. The sparse view acquisition method used in the present application greatly reduces the camera requirements for acquisition, reduces the cost of equipment and data processing, and is more practical in the case of limited resources. The smaller amount of camera acquisition data allows the algorithm to process faster, simplifies data management and analysis processes, and improves processing efficiency. In the arrangement implementation process, sparse view acquisition is more flexible than dense camera arrays, making it easier to apply in different scenes and environments.

[0055] Step 203: Perform view super-resolution processing on each sparse low-resolution view image of each scene layer to obtain a dense high-resolution view image of each scene layer.

[0056] In the present application, the resolution of the sparse low-resolution view image is lower than that of the dense high-resolution view image, and the number of views in the sparse low-resolution view image is less than that in the dense high-resolution view image.

[0057] In an exemplary embodiment, since the sparse acquisition scheme is used to collect low-resolution view images in step 202 to reduce rendering time, it is necessary to process the sparse low-resolution view images to improve the view density and image resolution. Step 203 includes steps 401 to 402.

[0058] Step 401: For any scene layer, generate virtual views for the sparse low-resolution view image of the scene layer to obtain a dense low-resolution view image of the scene layer.

[0059] Specifically, according to the sparse low-resolution view image of the scene layer, a disparity map super-resolution network is used to generate virtual views to obtain a dense low-resolution view image of the scene layer. The disparity map super-resolution network is obtained by pre-training sparse low-resolution view sample images and corresponding dense low-resolution view sample images. The disparity map super-resolution network can predict the view information of unknown positions according to the view information of known positions. The sparse low-resolution view image is inputted, and the sparse low-resolution view image is processed to generate the view image information of other uncollected positions, thereby obtaining a dense low-resolution view image.

[0060] In step 202, the sparse acquisition strategy is used, which reduces the rendering requirements, but the low number of views may cause discontinuous image changes between views when displaying the light field, resulting in jumps that greatly affect the viewing effect and experience. Therefore, in this step, a disparity map super-resolution network (or virtual view generation network) is selected to generate virtual views.

[0061] The multi-view super-resolution (SR) technology based on the disparity map and the motion vector is a method of reconstructing a dense image by using multiple view images, a disparity map and motion vector information. This kind of image method is particularly suitable for low resolution with multiple views, such as stereo vision, multi-camera shooting scenes, and can use different view images to reconstruct a dense view image in combination with a motion vector estimation method. Among them, the view image provides scene information, especially depth information, and can represent the spatial difference between each view image, and the motion vector can estimate the temporal or spatial change between the view images, helping to capture the motion information in the dynamic scene. Therefore, the present application trains the disparity map super-resolution network by taking the sparse low-resolution view sample image as the input and the dense low-resolution view sample image as the reference image, so that the disparity map super-resolution network can predict the scene information at other view positions, and then a dense view image can be obtained.

[0062] Step 402, performing super-resolution processing on the dense low-resolution view image of the scene layer to obtain a dense high-resolution view image of the scene layer.

[0063] Specifically, according to the dense low-resolution view image of the scene layer, an image super-resolution network is used to perform super-resolution processing to obtain a dense high-resolution view image of the scene layer. The image super-resolution network is obtained by pre-training using a dense low-resolution view sample image and a corresponding dense high-resolution view sample image.

[0064] In order to speed up the acquisition speed of the camera and reduce the resources consumed by rendering the view image, the resolution set during acquisition is low. Therefore, in this step, the image super-resolution network is used to improve the resolution of the dense low-resolution image.

[0065] Super-resolution is a technique to restore a low-resolution image into a high-resolution image through an algorithm. It includes but is not limited to using Convolutional Neural Networks (CNN) or Generative Adversarial Networks (GAN) in deep learning. During the training process, the image super-resolution network learns the mapping relationship through low-resolution and high-resolution image pairs, and makes the dense low-resolution view image more and more close to the dense high-resolution view image through optimizing the loss function (such as mean square error, perceptual loss, etc.). The dense low-resolution view sample image is input, and the dense high-resolution view sample image is used as the reference image to train the image super-resolution network. The training process of the image super-resolution network is repeated tens of thousands of times on a high-performance computer until the image super-resolution network reliably outputs a high-fidelity, high-resolution image. Once the training is completed, the generation of light field images can be greatly accelerated, and the generation quality of light field images can be effectively improved.

[0066] In step 204, the dense high-resolution view images of the multiple scene layers are encoded and synthesized according to the display parameters of the light field display and the front-back positional relationship between the multiple scene layers, to obtain a dense view high-resolution light field encoding image.

[0067] In an exemplary embodiment, step 204 includes steps 501 to 502.

[0068] In step 501, the dense high-resolution view images of each scene layer are respectively light field encoded according to the display parameters of the light field display, to obtain a light field encoding image of each scene layer.

[0069] Since the light field display displays 3D information, the images at each view position need to be arranged on the display screen according to a certain rule, and after being split by the light control elements on the light field display, the restored view image can be observed at the correct view position.

[0070] In step 502, the contents in the light field encoding images of the multiple scene layers are stacked according to the front-back positional relationship between the multiple scene layers, to obtain a dense view high-resolution light field encoding image.

[0071] Specifically, when the human eye is at different view positions, the human eye can observe a 3D scene with a three-dimensional stereoscopic effect due to the parallax between the two eyes. After the dense high-resolution view images of each scene layer are light field encoded, the scene content close to the camera is overlaid on the scene content far from the camera according to the front-back positional relationship between the scene layers, so as to obtain a dense view high-resolution light field encoding image with correct depth relationship.

[0072] Step 205, display the dense view point high resolution light field encoding image on the light field display. Further, a high-quality large-scale 3D scene can be displayed on the light field display.

[0073] The present application is applicable to a light field display and can realize high-quality large-scale 3D scene display. By using the hierarchical resource optimization and redundancy reduction method, the detail level and adaptive grid slicing technology, and the real-time optimization of scenes far from the zero plane, the consumption of computing resources can be significantly reduced while maintaining high image quality and visual immersion, and the redundant data in the rendering process is reduced, thereby realizing more efficient light field image generation, improving response speed and user experience, bringing new breakthroughs to the adaptive light field display system, making it perform better in complex scenes, and meeting the higher demand of users for realism and interactivity. The application of hierarchical variable resolution sampling technology is the key to realizing real-time generation of large-scale high-fidelity 3D light field content and promoting the rapid development of related fields. It provides users with a more smooth and immersive visual experience, meets the increasing demand for interaction, and lays a solid foundation for virtual reality, augmented reality and other frontier applications.

[0074] The present application also provides an application scenario of the three-dimensional scene display method. Specifically, the three-dimensional scene display method provided by the embodiment can be applied to three-dimensional city landscape scene display, and the specific process is as follows.

[0075] (1) Scene layering and camera deployment.

[0076] In the embodiment, a three-dimensional city landscape scene is displayed on a three-dimensional light field display with a horizontal viewing angle of 100 degrees. According to the in-and-out screen depth of the object, the scene is divided into five levels:

[0077] ① Zero plane layer: the part of the scene located at the zero plane position, and the part of the scene within 5cm in and out of the screen. 20 acquisition cameras are deployed within the 100-degree viewing range, and the camera resolution is set to 2k.

[0078] ② Close to zero plane layer: the scene part with an in-and-out screen depth of 5cm to 10cm, wherein the out-of-screen 5cm to 10cm and the in-of-screen 5cm to 10cm are two scene layers. 15 acquisition cameras are deployed within the 100-degree viewing range, the camera resolution is set to 1k, and the grid density of the scene layer is adjusted to be lower, occupying less resources.

[0079] ③ Far from zero plane layer: the scene with the depth of 10cm to 15cm, which is divided into two layers, the out-screen 10cm to 15cm and the in-screen 10cm to 15cm. 10 cameras are deployed within the 100-degree viewing range, the resolution of the deployed cameras is set to 720p, and the grid density of this layer is adjusted to be low, occupying less resources.

[0080] The final output is as follows:

[0081] Zero plane layer: 2k resolution parallax map 20.

[0082] Near zero plane layer: 1k resolution parallax map 15.

[0083] Far from zero plane layer: 720p resolution parallax map 10.

[0084] (2) Viewpoint super-resolution processing part.

[0085] ① Virtual viewpoint generation processing: calculate the motion vector using the sparse low-resolution viewpoint image, and use the sparse low-resolution viewpoint image and the motion vector as the input of the parallax map super-resolution network to generate the virtual viewpoint.

[0086] The results are as follows:

[0087] Zero plane layer: 2k resolution parallax map 50.

[0088] Near zero plane layer: 1k resolution parallax map 50.

[0089] Far from zero plane layer: 720p resolution parallax map 50.

[0090] ② Resolution super-resolution processing: input the generated dense low-resolution viewpoint image into the convolutional neural network for super-resolution processing to improve to 4K or 8K level.

[0091] The results are as follows:

[0092] Zero plane layer: 4k resolution parallax map 50.

[0093] Near zero plane layer: 4k resolution parallax map 50.

[0094] Far from zero plane layer: 4k resolution parallax map 50.

[0095] After two super-resolution networks, the dense high-resolution viewpoint image of each scene layer can be obtained.

[0096] (3) Light field encoding and synthesis.

[0097] According to the display characteristics of the three-dimensional light field display, sub-pixels in the dense high-resolution view point image are pixel-encoded to obtain a high-resolution light field encoding image. According to the front-back position relationship among the five scene layers, the pixels in the five light field encoding images are recolored, so that the scene content at the front position (closer to the human eye) covers the scene content at the back position, thereby obtaining a high-resolution light field encoding image with correct depth relationship display and dense view points. The three-dimensional light field display is a cylindrical lens grating three-dimensional display with a viewing angle of 100 degrees.

[0098] The application can generate high-quality three-dimensional visual effects in real time in a large-scale scene, and the consumption of data quantity and computing resources is effectively controlled. Not only can light field content be quickly generated, but also real-time display effects with clear details and accurate parallax can be obtained during multi-view dynamic display, thereby bringing a real immersive feeling to users. Compared with traditional methods, the consumption of computing resources is reduced, and the resource utilization efficiency is significantly improved, especially in the collection of long-range and simple areas.

[0099] Based on the same inventive concept, the embodiments of the application also provide a three-dimensional scene display system for implementing the three-dimensional scene display method described above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more three-dimensional scene display system embodiments provided below can refer to the limitations of the three-dimensional scene display method described above, which will not be described here.

[0100] In an exemplary embodiment, as shown in Figure 6 a three-dimensional scene display system is provided, which includes a scene layering module 601, a sparse view point collection module 602, a view point super-resolution module 603, an encoding and synthesis module 604, and a display module 605.

[0101] The scene layering module 601 is configured to layer the three-dimensional scene according to the distance of each region in the three-dimensional scene from the zero plane, and to reduce the redundancy of the content resources of the long-distance layer to obtain a plurality of scene layers. The long-distance layer is a scene layer with a distance from the zero plane greater than a set threshold.

[0102] The sparse view point collection module 602 is configured to collect sparse view points for each scene layer based on the distance of the scene layer from the zero plane to obtain a sparse low-resolution view point image of each scene layer.

[0103] The view point super-resolution module 603 is configured to perform view point super-resolution processing on the sparse low-resolution view point image of each scene layer to obtain a dense high-resolution view point image of each scene layer.

[0104] The coding and synthesizing module 604 is configured to code and synthesize the dense high-resolution viewpoint images of the plurality of scene layers according to the display parameters of the light field display and the front-back positional relationship between the plurality of scene layers, to obtain a dense viewpoint high-resolution light field coding image.

[0105] The display module 605 is configured to display the dense viewpoint high-resolution light field coding image on the light field display.

[0106] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 7 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store a three-dimensional scene. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a three-dimensional scene display method.

[0107] Those skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Figure 7 In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0108] In an exemplary embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0109]

[0110] ​It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0111] In the present application, all actions of obtaining signals, information or data are carried out in compliance with the data protection regulations and policies of the country where the device is located, and with the authorization of the owner of the corresponding device.

[0112] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0113] The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on blockchain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0114] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, it should be understood that the application encompasses all possible combinations of the technical features described above.

[0115] The principles and implementation manners of the present application are described herein by using specific examples, and the above embodiments are only used to help understand the method of the present application and its core idea; meanwhile, according to the idea of the present application, the specific implementation manners and application scopes will be changed by those skilled in the art. In conclusion, the content of the present specification should not be understood as a limitation of the present application.

Claims

1. A three-dimensional scene display method, characterized in that: The three-dimensional scene display method comprises: The 3D scene is layered according to the distance between each area in the 3D scene and the zero plane, and the content resources of the distant layer are reduced in redundancy to obtain multiple scene layers; the distant layer is the scene layer whose distance from the zero plane is greater than a set threshold; the content resources include grid density; Based on the distance between each scene layer and the zero plane, sparse viewpoint acquisition is performed on each scene layer to obtain a sparse low-resolution viewpoint image of each scene layer, specifically comprising: for any scene layer, determining the number and resolution of cameras for the scene layer according to the distance between the scene layer and the zero plane; the closer the scene layer is to the zero plane, the more cameras it has and the higher the resolution; based on the content resources of the scene layer, determining the viewpoint position of the scene layer that can capture the most scene information; and acquiring a sparse low-resolution viewpoint image of the scene layer based on the number, resolution, and viewpoint position of the scene layer; performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer respectively to obtain a dense high-resolution viewpoint image of each scene layer; the resolution of the sparse low-resolution viewpoint image is smaller than the resolution of the dense high-resolution viewpoint image, and the number of viewpoints in the sparse low-resolution viewpoint image is smaller than the number of viewpoints in the dense high-resolution viewpoint image; According to the display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, the dense high-resolution viewpoint images of the multiple scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field coding map; The dense viewpoint high-resolution light field coding image is displayed on the light field display.

2. The three-dimensional scene display method according to claim 1, characterized in that: The closer the scene layer is to the zero plane, the more cameras it has and the higher the resolution.

3. The three-dimensional scene display method according to claim 1, characterized in that: The sparse low-resolution viewpoint images of each scene layer are subjected to viewpoint super-resolution processing to obtain dense high-resolution viewpoint images of each scene layer, specifically including: For any scene layer, performing virtual viewpoint generation on the sparse low-resolution viewpoint image of the scene layer to obtain a dense low-resolution viewpoint image of the scene layer; Super-resolution processing is performed on the dense low-resolution viewpoint image of the scene layer to obtain a dense high-resolution viewpoint image of the scene layer.

4. The three-dimensional scene display method according to claim 3, characterized in that: Generating a virtual viewpoint on the sparse low-resolution viewpoint image of the scene layer to obtain a dense low-resolution viewpoint image of the scene layer specifically includes: According to the sparse low-resolution viewpoint image of the scene layer, a disparity map super-resolution network is used to generate virtual viewpoints to obtain a dense low-resolution viewpoint image of the scene layer; the disparity map super-resolution network is pre-trained using sparse low-resolution viewpoint sample images and corresponding dense low-resolution viewpoint sample images.

5. The three-dimensional scene display method according to claim 3, characterized in that: Performing super-resolution processing on the dense low-resolution viewpoint image of the scene layer to obtain a dense high-resolution viewpoint image of the scene layer specifically includes: According to the dense low-resolution viewpoint image of the scene layer, super-resolution processing is performed using an image super-resolution network to obtain a dense high-resolution viewpoint image of the scene layer; the image super-resolution network is pre-trained using dense low-resolution viewpoint sample images and corresponding dense high-resolution viewpoint sample images.

6. The three-dimensional scene display method according to claim 1, characterized in that: Based on the display parameters of the light field display and the front-to-back positional relationship between multiple scene layers, dense high-resolution viewpoint images of multiple scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field coding map, which specifically includes: performing light field encoding on the dense high-resolution viewpoint images of each scene layer according to the display parameters of the light field display to obtain a light field encoded image of each scene layer; According to the front-to-back positional relationship between multiple scene layers, the contents in the light field coding images of multiple scene layers are stacked to obtain a dense viewpoint high-resolution light field coding image.

7. A three-dimensional scene display system, applied to the three-dimensional scene display method according to any one of claims 1 to 6, characterized in that: The three-dimensional scene display system includes: A scene layering module is configured to layer the 3D scene according to the distance between each area in the 3D scene and the zero plane, and reduce the redundancy of content resources in the distant layers to obtain multiple scene layers; the distant layers are scene layers whose distance from the zero plane is greater than a set threshold; A sparse viewpoint acquisition module is used to perform sparse viewpoint acquisition on each scene layer based on the distance between each scene layer and the zero plane, and obtain a sparse low-resolution viewpoint image of each scene layer; The viewpoint super-resolution module is used to perform viewpoint super-resolution processing on the sparse low-resolution viewpoint images of each scene layer to obtain dense high-resolution viewpoint images of each scene layer; A coding and synthesis module is used to encode and synthesize dense high-resolution viewpoint images of multiple scene layers according to the display parameters of the light field display and the front-back position relationship between the multiple scene layers to obtain a dense viewpoint high-resolution light field coding map; A display module is configured to display the dense viewpoint high-resolution light field coding image on the light field display.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene display method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional scene display method according to any one of claims 1 to 6 is implemented.

Images