Three-dimensional scene display method, system, equipment and medium

Abstract

The invention discloses a three-dimensional scene display method, system and device and a medium, and relates to the field of display, and the method comprises the steps: carrying out the layering of a three-dimensional scene according to the distance between each region in the three-dimensional scene and a zero plane, and carrying out the content resource redundancy reduction of a long-distance layer; based on the distance between each scene layer and the zero plane, sparse viewpoint collection is carried out on each scene layer, and a sparse low-resolution viewpoint image of each scene layer is obtained; performing viewpoint super-division processing on the sparse low-resolution viewpoint image of each scene layer to obtain a dense high-resolution viewpoint image of each scene layer; and according to the display parameters of the light field display and the front-back position relationship among the plurality of scene layers, encoding and synthesizing the dense high-resolution viewpoint image to obtain a dense viewpoint high-resolution light field encoding graph, and displaying the dense viewpoint high-resolution light field encoding graph on the light field display. According to the invention, the consumption of data volume and computing resources is reduced, and a high-quality three-dimensional visual effect can be generated in real time in a large-scale scene.

Description

Technical Field

[0001] The present application relates to the field of display, and in particular to a three-dimensional scene display method, system, device and medium. Background Art

[0002] Light field images are usually composed of tens to tens of thousands of viewpoint images. This highly complex structure means that when performing large-scale scene light field rendering, the amount of calculation required is extremely large and complex, making real-time generation very difficult. Not only does each viewpoint image need to be processed independently, but the relationship and subtle changes between viewpoints also need to be considered, which further increases the computational burden. In addition, factors such as lighting, shadows, and object motion in large-scale scenes can also have a significant impact on the rendering quality, resulting in reduced processing efficiency. Therefore, how to reduce computational complexity while maintaining high image quality has become a major challenge in current light field rendering research. Summary of the invention

[0003] The purpose of this application is to provide a three-dimensional scene display method, system, device and medium, which can reduce the amount of data and the consumption of computing resources, and generate high-quality three-dimensional visual effects in real time in large-scale scenes.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a three-dimensional scene display method, comprising:

[0006] According to the distance between each area in the three-dimensional scene and the zero plane, the three-dimensional scene is layered, and the content resource redundancy of the long-distance layer is reduced to obtain multiple scene layers; the long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold;

[0007] Based on the distance between each scene layer and the zero plane, sparse viewpoints are collected for each scene layer to obtain a sparse low-resolution viewpoint image of each scene layer;

[0008] Performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer respectively to obtain a dense high-resolution viewpoint image of each scene layer; the resolution of the sparse low-resolution viewpoint image is smaller than the resolution of the dense high-resolution viewpoint image, and the number of viewpoints in the sparse low-resolution viewpoint image is smaller than the number of viewpoints in the dense high-resolution viewpoint image;

[0009] According to display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, dense high-resolution viewpoint images of the multiple scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field encoding image;

[0010] The dense viewpoint high-resolution light field encoding image is displayed on the light field display.

[0011] In a second aspect, the present application provides a three-dimensional scene display system, comprising:

[0012] A scene layering module is used to layer the three-dimensional scene according to the distance between each area in the three-dimensional scene and the zero plane, and reduce the redundancy of content resources in the long-distance layer to obtain multiple scene layers; the long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold;

[0013] A sparse viewpoint acquisition module is used to perform sparse viewpoint acquisition on each scene layer based on the distance between each scene layer and the zero plane, so as to obtain a sparse low-resolution viewpoint image of each scene layer;

[0014] A viewpoint super-resolution module is used to perform viewpoint super-resolution processing on the sparse low-resolution viewpoint images of each scene layer to obtain a dense high-resolution viewpoint image of each scene layer;

[0015] A coding and synthesis module, used to encode and synthesize dense high-resolution viewpoint images of multiple scene layers according to display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, to obtain a dense viewpoint high-resolution light field coding image;

[0016] A display module is used to display the dense viewpoint high-resolution light field coding image on the light field display.

[0017] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned three-dimensional scene display method.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned three-dimensional scene display method when executed by a processor.

[0019] According to the specific embodiments provided in this application, this application has the following technical effects:

[0020] The present application provides a three-dimensional scene display method, system, device and medium, which layers the three-dimensional scene according to the distance between each area in the three-dimensional scene and the zero plane, and reduces the redundancy of content resources in the distant layer, thereby reducing redundant data in the rendering process, and sparse viewpoint collection is performed on each scene layer based on the distance between each scene layer and the zero plane, thereby reducing the amount of data and the consumption of computing resources, and by performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image, it is possible to have a real-time display effect with clear details and accurate parallax during multi-view dynamic display. Therefore, the present application can generate high-quality three-dimensional visual effects in real time in large-scale scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 This is an application environment diagram of a three-dimensional scene display method in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the overall process of a three-dimensional scene display method provided in one embodiment of the present application;

[0024] Figure 3 A detailed flowchart of a three-dimensional scene display method provided in one embodiment of the present application;

[0025] Figure 4 Schematic diagram of the generation process of a dense viewpoint high-resolution light field coding map in one embodiment of the present application;

[0026] Figure 5 This is a schematic diagram of grid slicing grouping in one embodiment of the present application;

[0027] Figure 6 A schematic diagram of functional modules of a three-dimensional scene display system provided in one embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In order to solve the problem of performance degradation in adaptive naked-eye 3D light field display, it is urgent to conduct in-depth research on layered variable resolution sampling technology. Layered variable resolution sampling technology aims to reduce redundant viewpoint image information by dynamically adjusting the sampling density of different viewpoints, thereby significantly optimizing rendering efficiency.

[0031] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] The three-dimensional scene display method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the three-dimensional scene to be displayed to the server 104. After the server 104 receives the three-dimensional scene to be displayed, it generates a dense viewpoint high-resolution light field coding map and displays it on the light field display. In addition, in some embodiments, the three-dimensional scene display method can also be implemented by the server 104 or the terminal 102 alone.

[0033] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0034] In an exemplary embodiment, Figures 2 to 4 As shown, a three-dimensional scene display method is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 205.

[0035] Step 201, according to the distance between each area in the three-dimensional scene and the zero plane, the three-dimensional scene is layered, and the content resource redundancy of the long-distance layer is reduced to obtain multiple scene layers. The long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold.

[0036] Determine the screen entry and exit distances of each part of the 3D scene when it is displayed on the light field display. According to the screen entry and exit distances of the scene and the display depth characteristics of the current light field display, the 3D scene is layered, and the resource redundancy of the scene layer far from the zero plane is reduced.

[0037] The layering method and the number of scene layers can be flexibly determined according to the display effect and display characteristics of the light field display. In 3D light field display, due to the inherent characteristics of light field display, the area with the best visual effect is the area close to the zero plane. The greater the distance between the screen and the zero plane, that is, the greater the distance between the screen and the zero plane, the more blurred the visual information and the worse the visual effect. Different three-dimensional stereo displays have different display depth of field ranges. Therefore, before layering the scene according to the visual information, the depth of field range of the display and the visual effect of the display area at different depths of field should be measured. The judgment of the visual effect can be quantified using objective image quality evaluation indicators including but not limited to Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). It can also be based on the subjective feelings of the viewer and the evaluated index value as the basis for scene layering.

[0038] In an exemplary embodiment, the scene is finely layered according to the display depth range of the three-dimensional light field display and the visual display effects at each display depth, and the scene is divided into multiple levels, each of which corresponds to a different area in the scene. The importance of the scene is determined by its distance from the zero plane. The closer to the zero plane, the higher the importance. The higher the fineness of the scene when displayed, the better the visual effect. The entry and exit screen areas that are at and near the zero plane and have low crosstalk when displayed on the light field display and have high visual display quality are grouped together; the entry and exit screen scene areas that are far from the zero plane and have slightly poor visual display quality are grouped together; the entry and exit screen scene areas that are far from the zero plane and have very poor visual display effects are grouped together.

[0039] After scene layering, resources are optimized for each scene layer, especially for the scene layers that are far away from the zero plane.

[0040] 3D scene content resources are composed of basic primitive meshes, materials, and ambient lighting information. In order to improve the display performance of 3D light fields, the mesh structure and central camera parameters of the 3D virtual scene are used as input, and the Levels of Detail (LOD) structure is constructed through mesh preprocessing to optimize the content resources. Specifically, the basic principle of light field content resource optimization is to allocate more mesh faces to parts with strong 3D light field display performance, and fewer mesh faces to parts with weaker performance. This can effectively reduce information redundancy and reduce the computational pressure during content generation.

[0041] This application uses LOD and adaptive grid slicing technology to pre-process the high-precision scene model into an adaptive grid body, and adjust the grid density in real time according to the distance between the scene and the zero plane to optimize the rendering effect. The grid density of the scene close to the zero plane is high, and the display accuracy is high. The grid density of the scene far from the zero plane is low, and the display accuracy is low. Therefore, according to the display characteristics of the light field display, multi-level optimization of the layered scene is achieved, the rendering consumption of the scene far from the zero plane is reduced, and computing resources are saved.

[0042] Since high-precision images often contain redundant information, adaptive grid slicing technology can effectively merge these redundant grids and reduce the amount of computation consumed by the camera during acquisition (rendering). Figure 5 As shown, mesh fragmentation divides the scene model into multiple fragments with the same number of triangles (usually 64×64), which helps to balance details and performance and enables the scene to be quickly adjusted at different viewing distances and perspectives. Figure 5 The left side of the figure is a schematic diagram of grid slicing, and the right side is a schematic diagram of composing 18 slices into a slice group.

[0043] Step 202 : Based on the distance between each scene layer and the zero plane, sparse viewpoint acquisition is performed on each scene layer to obtain a sparse low-resolution viewpoint image of each scene layer.

[0044] In an exemplary embodiment, step 202 includes the following steps 301 to 303 .

[0045] Step 301: for any scene layer, determine the number of cameras and the resolution of the scene layer according to the distance between the scene layer and the zero plane. The closer the scene layer is to the zero plane, the more cameras it has and the higher the resolution.

[0046] This application flexibly sets the number and resolution of cameras in each scene layer according to the distance between each scene layer and the zero plane to ensure the best visual effect and computing efficiency. When processing complex scenes, it can still maintain good performance and avoid delays or freezes due to excessive resource consumption. Through flexible configuration strategies, not only the efficiency of information collection is significantly improved, so that information in important areas can be fully captured, but also resource consumption is effectively reduced, so that higher response speeds and excellent image quality can be achieved in real-time applications. The advantage of layered design is also reflected in its adaptability. With the development of technology and changes in application needs, the camera configuration and resolution settings of each layer can be dynamically adjusted according to specific scene characteristics and user needs.

[0047] The scene layer close to the zero plane has high display quality on the light field display and little crosstalk, so a large number of cameras and high resolution are selected to capture rich details and dynamic information, ensuring that the audience can feel the realism and delicacy of the picture when watching. It not only helps to show the texture and light and shadow changes of objects, but also enhances the audience's immersive experience, making the interaction smoother and more natural.

[0048] The scene layer far away from the zero plane has a large screen entry and exit distance and a large crosstalk effect, resulting in poor visual effects and relatively low requirements for details. Therefore, a smaller number of cameras and a lower resolution can be selected, which can effectively reduce the computing burden, optimize resource utilization, and save rendering resources.

[0049] Step 302: Determine the viewpoint position of the scene layer according to the content resources of the scene layer.

[0050] In an exemplary embodiment, more densely packed cameras are placed in complex scene locations for acquisition, and more sparsely packed cameras are placed in simple locations, thereby arranging a smaller number of cameras at viewpoints that can capture more scene information, thereby minimizing the redundancy of acquired information and the consumption of acquisition time.

[0051] This application uses a sparse acquisition strategy based on the viewpoint measurement method to select the appropriate viewpoint position by analyzing the geometric features of the 3D scene model and calculating the importance of the vertices to efficiently collect scene information. Since the 3D scene model is generally expressed by fitted geometric patches, and the geometric content is also an important aspect of the perception model, the viewpoint measurement method measures the amount of observable information at different positions by observing and detecting the geometric features of the 3D scene, with the aim of selecting viewpoints that contain more information, thereby achieving efficient collection of scene information.

[0052] Among them, the sparse acquisition strategy based on the viewpoint measurement method aims to find the viewpoint position that can collect the most scene information, so as to achieve more scene information acquisition with fewer viewpoints. Specifically, the viewpoint entropy method can be used to calculate the Gaussian weighted curvature of each vertex in the three-dimensional scene model at multiple scales, and the importance of the vertex can be measured based on this. The sum of the importance of the visible vertices of each viewpoint is then used to measure the quality of the viewpoint, so as to select the appropriate sparse acquisition position. Other viewpoint measurement methods can also be used to select sparse acquisition positions.

[0053] Step 303: Based on the number of cameras, resolution and viewpoint positions of the scene layer, a sparse low-resolution viewpoint image of the scene layer is collected.

[0054] Specifically, a sparse camera arrangement is used to perform low-resolution acquisition of the scene layer to obtain a sparse low-resolution viewpoint image. The sparse viewpoint acquisition method used in this application greatly reduces the camera requirements for acquisition, reduces equipment costs and the computational cost of data processing, and is more practical when resources are limited. The smaller amount of data collected by the camera allows the algorithm to speed up processing, simplify data management and analysis processes, and improve processing efficiency. In the process of layout implementation, sparse viewpoint acquisition is more flexible than dense camera arrays, and is easy to apply in different scenes and environments.

[0055] Step 203 , performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer respectively, to obtain a dense high-resolution viewpoint image of each scene layer.

[0056] The resolution of the sparse low-resolution viewpoint image is smaller than that of the dense high-resolution viewpoint image, and the number of viewpoints in the sparse low-resolution viewpoint image is smaller than the number of viewpoints in the dense high-resolution viewpoint image.

[0057] In an exemplary embodiment, since a sparse acquisition scheme is used to acquire low-resolution viewpoint images in order to reduce rendering time in step 202, it is necessary to process the sparse low-resolution viewpoint images to improve viewpoint density and image resolution. Step 203 includes the following steps 401 to 402.

[0058] Step 401 : for any scene layer, generate a virtual viewpoint for a sparse low-resolution viewpoint image of the scene layer to obtain a dense low-resolution viewpoint image of the scene layer.

[0059] Specifically, according to the sparse low-resolution viewpoint image of the scene layer, a disparity map super-resolution network is used to generate virtual viewpoints to obtain a dense low-resolution viewpoint image of the scene layer. The disparity map super-resolution network is pre-trained using sparse low-resolution viewpoint sample images and corresponding dense low-resolution viewpoint sample images. The disparity map super-resolution network can predict viewpoint information of unknown positions based on viewpoint information of known positions, take the sparse low-resolution viewpoint image as input, process the sparse low-resolution viewpoint image, and generate viewpoint image information of other uncollected positions, thereby obtaining a dense low-resolution viewpoint image.

[0060] The sparse acquisition strategy used in step 202 reduces the rendering requirements, but the low number of viewpoints will cause discontinuous image changes between viewpoints during light field display, resulting in jumps, which greatly affects the viewing effect and experience. For this reason, a disparity map super-resolution network (or virtual viewpoint generation network) is selected in this step to realize the generation of virtual viewpoints.

[0061] Multi-view super-resolution (SR) technology based on disparity map and motion vector is a method to reconstruct dense images through multiple view images, disparity map and motion vector information. This type of image method is particularly suitable for low-resolution with multiple view angles, such as stereoscopic vision and scenes shot by multiple cameras. Different view images can be used in combination with motion vector estimation methods to reconstruct dense viewpoint images. Among them, the viewpoint image provides scene information, especially depth information, and can represent the spatial difference between each view image. The motion vector can estimate the time or space changes between view images, helping to capture motion information in dynamic scenes. Therefore, the present application uses sparse low-resolution viewpoint sample images as input and dense low-resolution viewpoint sample images as reference images to train the disparity map super-resolution network, which can enable the disparity map super-resolution network to predict the scene information at other viewpoint positions, and then obtain dense viewpoint images.

[0062] Step 402: Perform super-resolution processing on the dense low-resolution viewpoint image of the scene layer to obtain a dense high-resolution viewpoint image of the scene layer.

[0063] Specifically, according to the dense low-resolution viewpoint image of the scene layer, an image super-resolution network is used to perform super-resolution processing to obtain a dense high-resolution viewpoint image of the scene layer. The image super-resolution network is pre-trained using dense low-resolution viewpoint sample images and corresponding dense high-resolution viewpoint sample images.

[0064] In order to speed up the camera acquisition and reduce the resources consumed in rendering viewpoint images, the resolution is set lower during acquisition. Therefore, an image super-resolution network is used in this step to improve the resolution of dense low-resolution images.

[0065] Super-resolution is a technique that uses algorithms to restore low-resolution images to high-resolution images. This includes but is not limited to the use of convolutional neural networks (CNN) or generative adversarial networks (GAN) in deep learning. During the training process, the image super-resolution network learns the mapping relationship between low-resolution and high-resolution image pairs, and optimizes the loss function (such as mean square error, perceptual loss, etc.) to make dense low-resolution viewpoint images closer and closer to dense high-resolution viewpoint images. The dense low-resolution viewpoint sample images are used as input, and the dense high-resolution viewpoint sample images are used as reference images to train the image super-resolution network. The training process of the image super-resolution network is repeated tens of thousands of times on high-performance computers until the image super-resolution network reliably outputs highly realistic and high-resolution images. Once the training is completed, the generation of light field images can be greatly accelerated, and the generation quality of light field images can be effectively improved.

[0066] Step 204 , encoding and synthesizing dense high-resolution viewpoint images of multiple scene layers according to display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, to obtain a dense viewpoint high-resolution light field coding image.

[0067] In an exemplary embodiment, step 204 includes the following steps 501 to 502 .

[0068] Step 501 : performing light field coding on dense high-resolution viewpoint images of each scene layer according to display parameters of the light field display to obtain a light field coded image of each scene layer.

[0069] Since the light field display shows 3D information, the images at each viewpoint position need to be arranged on the display screen according to a certain pattern. After the light is split by the light control element on the light field display, the restored viewpoint image can be seen at the correct viewpoint position.

[0070] Step 502 : stacking the contents of the light field coding images of the multiple scene layers according to the front-to-back position relationship between the multiple scene layers to obtain a dense viewpoint high-resolution light field coding image.

[0071] Specifically, when the human eye is at different viewpoints, due to the parallax between the two eyes, the human eye can observe a 3D scene with a three-dimensional effect. After the dense high-resolution viewpoint image of each scene layer is light field encoded, the scene content close to the camera covers the scene content far from the camera according to the front-to-back position relationship between the scene layers, thereby obtaining a high-resolution light field encoding image with a correct depth relationship and dense viewpoint.

[0072] Step 205: Display the dense viewpoint high-resolution light field coding image on the light field display, so that a high-quality large-scale 3D scene can be displayed on the light field display.

[0073] This application is applicable to light field displays and can achieve high-quality large-scale 3D scene display. Through layered resource optimization and redundancy reduction methods, using detail levels and adaptive grid slicing technology, scenes far from the zero plane are optimized in real time, which can significantly reduce the consumption of computing resources while maintaining high image quality and visual immersion, while reducing redundant data in the rendering process, thereby achieving more efficient light field image generation, and can improve response speed and user experience, bringing new breakthroughs to adaptive light field display systems, making their performance in complex scenes better, and meeting users' higher demands for realism and interactivity. Combined with the application of layered variable resolution sampling technology, it is the key to achieving large-scale real-time generation of high-fidelity 3D light field content and promoting the rapid development of related fields. Provide users with a smoother and more immersive visual experience, meet the ever-increasing interactive needs, and lay a solid foundation for cutting-edge applications such as virtual reality and augmented reality.

[0074] The present application also provides an application scenario, which applies the above-mentioned three-dimensional scene display method. Specifically: the three-dimensional scene display method provided in this embodiment can be applied to the three-dimensional urban landscape scene display, and the specific process is as follows.

[0075] (1) Scene layering and camera deployment.

[0076] In this embodiment, a three-dimensional urban landscape scene is designed to be displayed on a three-dimensional light field display with a horizontal viewing angle of 100 degrees, and is divided into five levels according to the depth of objects entering and exiting the screen:

[0077] ① Zero plane layer: the part of the scene at the zero plane position, and the part of the scene within 5cm of the screen. 20 acquisition cameras are deployed within a viewing range of 100 degrees, and the camera resolution is set to 2k.

[0078] ② Layer close to zero plane: The scene part with a screen depth of 5cm to 10cm, of which 5cm to 10cm out of the screen and 5cm to 10cm into the screen are two scene layers. 15 acquisition cameras are deployed within a viewing range of 100 degrees, and the camera resolution is set to 1k. The grid density of this scene layer is adjusted to make it lower in density and occupy less resources.

[0079] ③ Far from zero plane layer: scenes with a screen depth of 10cm to 15cm, where the screen depth of 10cm to 15cm and the screen depth of 10cm to 15cm are two scene layers. Deploy 10 acquisition cameras within a 100-degree viewing range, set the resolution of the deployed cameras to 720p, and adjust the grid density of this scene layer to make it lower in density and occupy less resources.

[0080] Finally, the following output is obtained:

[0081] The layer where the zero plane is located: 20 2k resolution disparity maps.

[0082] Close to the zero plane layer: 15 1k resolution disparity maps.

[0083] Out of the zero plane layer: 10 disparity maps with 720p resolution.

[0084] (2) Viewpoint super-resolution processing part.

[0085] ① Virtual viewpoint generation processing: Use sparse low-resolution viewpoint images to calculate motion vectors, and use the sparse low-resolution viewpoint images and motion vectors as input to the disparity map super-resolution network to generate virtual viewpoints.

[0086] The results are as follows:

[0087] Zero plane layer: 50 2k resolution disparity maps.

[0088] Close to the zero plane layer: 50 1k resolution disparity maps.

[0089] Away from the zero plane layer: 50 disparity maps at 720p resolution.

[0090] ② Resolution super-resolution processing: The generated dense low-resolution viewpoint image is used as input and super-resolution processed through a convolutional neural network to upgrade it to 4K or 8K level.

[0091] The following results are obtained:

[0092] Zero plane layer: 50 4k resolution disparity maps.

[0093] Close to the zero plane layer: 50 4k resolution disparity maps.

[0094] Out of the zero plane layer: 50 4k resolution disparity maps.

[0095] After two super-resolution networks, dense high-resolution viewpoint images of each scene layer can be obtained.

[0096] (3) Light field coding and synthesis.

[0097] According to the display characteristics of the three-dimensional light field display, the sub-pixels in the dense high-resolution viewpoint image are pixel-encoded to obtain a high-resolution light field coded image. According to the front-to-back position relationship between the five scene layers, the pixels in the five light field coded images are recolored so that the scene content at the front position (closer to the human eye) covers the scene content at the back position, thereby obtaining a high-resolution light field coded image with correct depth relationship display and dense viewpoints. Among them, the three-dimensional light field display is a cylindrical lens grating three-dimensional display with a viewing angle of 100 degrees.

[0098] This application can generate high-quality 3D visual effects in large-scale scenes in real time, and the consumption of data volume and computing resources is effectively controlled. Not only can it quickly generate light field content, but it can also have real-time display effects with clear details and accurate parallax during multi-view dynamic display, giving users a real sense of immersion. Compared with traditional methods, it reduces the consumption of computing resources, especially in the acquisition of distant views and simple areas, and the resource utilization efficiency has been significantly improved.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a three-dimensional scene display system for implementing the three-dimensional scene display method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more three-dimensional scene display system embodiments provided below can refer to the limitations of the three-dimensional scene display method above, and will not be repeated here.

[0100] In an exemplary embodiment, Figure 6 As shown, a three-dimensional scene display system is provided, including: a scene layering module 601, a sparse viewpoint acquisition module 602, a viewpoint super-resolution module 603, a coding synthesis module 604 and a display module 605.

[0101] The scene layering module 601 is used to layer the 3D scene according to the distance between each area in the 3D scene and the zero plane, and reduce the redundancy of content resources in the long-distance layer to obtain multiple scene layers. The long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold.

[0102] The sparse viewpoint acquisition module 602 is used to perform sparse viewpoint acquisition on each scene layer based on the distance between each scene layer and the zero plane, so as to obtain a sparse low-resolution viewpoint image of each scene layer.

[0103] The viewpoint super-resolution module 603 is used to perform viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer to obtain a dense high-resolution viewpoint image of each scene layer.

[0104] The encoding and synthesis module 604 is used to encode and synthesize dense high-resolution viewpoint images of multiple scene layers according to the display parameters of the light field display and the front-to-back position relationship between the multiple scene layers to obtain a dense viewpoint high-resolution light field coding image.

[0105] The display module 605 is used to display the dense viewpoint high-resolution light field coding image on the light field display.

[0106] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store three-dimensional scenes. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional scene display method is implemented.

[0107] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0109] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0110] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0111] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0113] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A three-dimensional scene display method, characterized in that: The three-dimensional scene display method comprises: According to the distance between each area in the three-dimensional scene and the zero plane, the three-dimensional scene is layered, and the content resource redundancy of the long-distance layer is reduced to obtain multiple scene layers; the long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold; Based on the distance between each scene layer and the zero plane, sparse viewpoints are collected for each scene layer to obtain a sparse low-resolution viewpoint image of each scene layer; Performing viewpoint super-resolution processing on the sparse low-resolution viewpoint image of each scene layer respectively to obtain a dense high-resolution viewpoint image of each scene layer; the resolution of the sparse low-resolution viewpoint image is smaller than the resolution of the dense high-resolution viewpoint image, and the number of viewpoints in the sparse low-resolution viewpoint image is smaller than the number of viewpoints in the dense high-resolution viewpoint image; According to display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, dense high-resolution viewpoint images of the multiple scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field encoding image; The dense viewpoint high-resolution light field encoding image is displayed on the light field display.

2. The three-dimensional scene display method according to claim 1, characterized in that: Based on the distance between each scene layer and the zero plane, sparse viewpoints are collected for each scene layer to obtain a sparse low-resolution viewpoint image of each scene layer, including: For any scene layer, determining the number and resolution of cameras of the scene layer according to the distance between the scene layer and the zero plane; Determining a viewpoint position of the scene layer according to content resources of the scene layer; Based on the number, resolution and viewpoint positions of the cameras of the scene layer, a sparse low-resolution viewpoint image of the scene layer is collected.

3. The three-dimensional scene display method according to claim 2, characterized in that: The closer the scene layer is to the zero plane, the more cameras it has and the higher the resolution.

4. The three-dimensional scene display method according to claim 1, characterized in that: The sparse low-resolution viewpoint images of each scene layer are respectively subjected to viewpoint super-resolution processing to obtain dense high-resolution viewpoint images of each scene layer, specifically including: For any scene layer, performing virtual viewpoint generation on a sparse low-resolution viewpoint image of the scene layer to obtain a dense low-resolution viewpoint image of the scene layer; Super-resolution processing is performed on the dense low-resolution viewpoint image of the scene layer to obtain a dense high-resolution viewpoint image of the scene layer.

5. The three-dimensional scene display method according to claim 4, characterized in that: Generating a virtual viewpoint for the sparse low-resolution viewpoint image of the scene layer to obtain a dense low-resolution viewpoint image of the scene layer specifically includes: According to the sparse low-resolution viewpoint image of the scene layer, a disparity map super-resolution network is used to generate virtual viewpoints to obtain a dense low-resolution viewpoint image of the scene layer; the disparity map super-resolution network is pre-trained using sparse low-resolution viewpoint sample images and corresponding dense low-resolution viewpoint sample images.

6. The three-dimensional scene display method according to claim 4, characterized in that: Performing super-resolution processing on the dense low-resolution viewpoint image of the scene layer to obtain a dense high-resolution viewpoint image of the scene layer specifically includes: According to the dense low-resolution viewpoint image of the scene layer, super-resolution processing is performed using an image super-resolution network to obtain a dense high-resolution viewpoint image of the scene layer; the image super-resolution network is pre-trained using dense low-resolution viewpoint sample images and corresponding dense high-resolution viewpoint sample images.

7. The three-dimensional scene display method according to claim 1, characterized in that: According to the display parameters of the light field display and the front-to-back position relationship between multiple scene layers, dense high-resolution viewpoint images of multiple scene layers are encoded and synthesized to obtain a dense viewpoint high-resolution light field coding map, which specifically includes: performing light field encoding on dense high-resolution viewpoint images of each scene layer according to display parameters of the light field display to obtain a light field encoded image of each scene layer; According to the front-to-back position relationship between multiple scene layers, the contents in the light field coding images of multiple scene layers are stacked to obtain a dense viewpoint high-resolution light field coding image.

8. A three-dimensional scene display system, applied to the three-dimensional scene display method according to any one of claims 1 to 7, characterized in that: The three-dimensional scene display system comprises: A scene layering module is used to layer the three-dimensional scene according to the distance between each area in the three-dimensional scene and the zero plane, and reduce the redundancy of content resources in the long-distance layer to obtain multiple scene layers; the long-distance layer is a scene layer whose distance from the zero plane is greater than a set threshold; A sparse viewpoint acquisition module is used to perform sparse viewpoint acquisition on each scene layer based on the distance between each scene layer and the zero plane, so as to obtain a sparse low-resolution viewpoint image of each scene layer; A viewpoint super-resolution module is used to perform viewpoint super-resolution processing on the sparse low-resolution viewpoint images of each scene layer to obtain a dense high-resolution viewpoint image of each scene layer; A coding and synthesis module, used to encode and synthesize dense high-resolution viewpoint images of multiple scene layers according to display parameters of the light field display and the front-to-back position relationship between the multiple scene layers, to obtain a dense viewpoint high-resolution light field coding image; A display module is used to display the dense viewpoint high-resolution light field coding image on the light field display.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene display method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional scene display method described in any one of claims 1 to 7 is implemented.

Images