Light field display optimization method and system based on deep learning

Abstract

The invention relates to a light field display optimization method and system based on deep learning. The method comprises the following steps: collecting a multi-key view angle sparse image covering a plurality of key view angles in a full-target view angle space by adopting a sparse sampling strategy; constructing a viewpoint super-resolution synthesis network based on deep learning; performing viewpoint interpolation and complementation on the multi-key-view-angle sparse image by using a viewpoint super-resolution synthesis network to generate a full-target-view-angle complete image; performing optical field synthesis on the full-target view angle complete image through image rendering, and mapping the image to a display screen through a macro pixel structure; the non-uniform microlens array is used for carrying out angle and direction modulation on image light displayed by the display screen, so that audiences can receive images in different directions at different observation positions. The display quality and the visual angle continuity of the light field image are remarkably improved, and the system performance is improved under the condition that hardware resources are limited.

Description

Technical Field

[0001] This invention relates to the field of light field display, and more specifically to a light field display optimization method and system based on deep learning. Background Technology

[0002] Light field display technology has received widespread attention in recent years, especially in fields such as virtual reality, augmented reality, and glasses-free 3D displays, where it has become an important means of providing immersive 3D visual experiences. Its basic principle is to simulate light from different perspectives to display multi-viewpoint images to users, thereby achieving realistic spatial perception and smooth motion parallax. However, as the number of viewpoints increases, light field display systems face a key bottleneck: the conflict between resolution and bandwidth. With display devices having more than 50 viewpoints, allocating sufficient resolution to each viewpoint will exceed the physical resolution of the display and the data transmission bandwidth required, leading to bandwidth overload and image quality degradation.

[0003] Currently, most light field display systems employ a uniform microlens array design to capture multi-viewpoint images. However, as the number of viewpoints increases, the system's bandwidth requirements also increase. If an independent high-resolution image is allocated to each viewpoint, the resolution of a single viewpoint will significantly decrease, typically below 1080p. Moreover, this allocation method greatly increases the bandwidth requirements for data transmission, leading to inefficient system operation. To balance the number of viewpoints and resolution, current light field display solutions typically compromise by reducing the number of viewpoints or lowering the resolution. However, this approach cannot meet the demands of glasses-free 3D display and high-quality light field rendering, affecting the smoothness of the display effect and the realism of visual perception. Summary of the Invention

[0004] This invention provides a light field display optimization method and system based on deep learning to solve at least one of the above-mentioned technical problems.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A light field display optimization method based on deep learning, comprising: A sparse sampling strategy was employed to acquire multi-key-view sparse images covering multiple key viewpoints in the full target viewpoint space. Construct a deep learning-based viewpoint super-resolution synthesis network; The viewpoint super-resolution synthesis network is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image to generate a complete image of the entire target viewpoint. The complete image from the full target viewpoint is synthesized using image rendering and then mapped onto the display screen through a macro-pixel structure. The angle and direction of the image light displayed on the screen are modulated by a non-uniform microlens array so that the viewer receives images from different directions at different viewing positions.

[0006] Based on the above technical solution, the present invention can be further improved as follows.

[0007] Furthermore, the non-uniform microlens array includes a central region, an edge region, and a transition region located between the central region and the edge region; the non-uniform microlens array is configured as follows: The central region of the non-uniform microlens array covers a field of view of ±20° and is arranged with high density microlenses, with each microlens projecting four viewpoints in different directions. The transition zone of the non-uniform microlens array covers a field of view of 20° to 30°, and is arranged with medium density microlenses, with each microlens projecting viewpoints in three directions. The edge region of the non-uniform microlens array covers a field of view greater than 30°, and a low-density microlens arrangement is used, with each microlens projecting viewpoints in two directions.

[0008] Furthermore, the viewpoint super-resolution synthesis network includes a feature extraction module, a disparity guidance module, a global Transformer module, and a multi-scale reconstruction module; The viewpoint super-resolution synthesis network is used to perform viewpoint interpolation and completion on the multi-key viewpoint images, specifically including: In the feature extraction module, a shared-weight 3D convolution is used to jointly extract multi-level features of the multi-key viewpoint image in both spatial and viewpoint dimensions to capture parallax changes and angular differences between viewpoints, thereby obtaining basic feature representations. In the disparity guidance module, based on the internally predicted disparity map, spatial distortion and weighted fusion are performed on the adjacent viewpoint features in the basic feature representation; In the global Transformer module, a self-attention mechanism is used to capture the long-range dependencies between non-neighboring viewpoint features in the basic feature representation; In the multi-scale reconstruction module, the outputs of the disparity guidance module and the global Transformer module are fused together, and viewpoint interpolation and completion are performed by combining a residual dense network and sub-pixel convolution, and super-resolution enhancement is performed to output a high-resolution complete image of the entire target viewpoint.

[0009] Furthermore, the process of acquiring the multi-key-view sparse image also includes: acquiring the view spatial position information corresponding to each key-view sparse image in the multi-key-view sparse image, and representing the view spatial position information as two-dimensional coordinates through a normalized mapping method to obtain a two-dimensional position code. The viewpoint super-resolution synthesis network is equipped with a position-aware guidance mechanism, which is used to assist the viewpoint super-resolution synthesis network in learning the spatial relationships between viewpoints based on the two-dimensional position encoding.

[0010] Furthermore, in the process of constructing the viewpoint super-resolution synthesis network, a differentiable rendering module and a composite loss function are introduced to perform end-to-end joint optimization learning training on the viewpoint super-resolution synthesis network; wherein: The differentiable rendering module is used to simulate brightness diffusion, directional blurring, and edge distortion during the optical projection process of a non-uniform microlens array. The composite loss function is expressed as follows: ; In the formula, , , , , These are composite loss, pixel error loss, perceptual loss, epipolar consistency loss, and adversarial loss, respectively. , , , These are the loss weights corresponding to pixel error loss, perceptual loss, epipolar consistency loss, and adversarial loss, respectively, and the loss weights are dynamically adjusted according to the features of different image regions during training.

[0011] Furthermore, before using the viewpoint super-resolution synthesis network to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image, the method further includes: pruning and optimizing the viewpoint super-resolution synthesis network and then deploying it to the image processing platform. Specifically, the viewpoint super-resolution synthesis network is pruned and optimized using model compression before being deployed to an image processing platform, including: Statistical analysis of the APoZ metric of the convolutional channels in the viewpoint super-resolution synthesis network across the entire training dataset; Convolutional channels with APoZ values ​​greater than a preset threshold are pruned and removed, and the remaining convolutional channels are quantized using 8-bit integers to obtain a viewpoint super-resolution synthesis network optimized by pruning and quantization compression. Deploy the pruned and quantized compression optimized viewpoint super-resolution synthesis network on the image processing platform; The quantization process is implemented using the following formula: ; In the formula, This is the result of 8-bit integer quantization. It has 32-bit floating-point convolution channels; It is a quantification factor, and ; It is the offset, and ;in, and These are the maximum and minimum values ​​of the quantization weights, respectively.

[0012] Furthermore, when synthesizing the light field of the complete image from the full target view through image rendering, the image rendering employs a pixel-level aggregation function, and the optical imaging model of the pixel-level aggregation function is expressed as follows: ; In the formula, , indicating the pixel position; This indicates the final image to be displayed; Indicates perspective At pixel position Projection weights at the location; Indicates perspective At pixel position The corresponding complete image of the entire target from all angles.

[0013] Furthermore, when synthesizing the complete image from the full target viewpoint using light field and mapping it to the display screen through a macro-pixel structure, the mapping structure includes: Each microlens in the non-uniform microlens array covers a corresponding macropixel in the display screen, and the macropixel is divided into multiple sub-pixel groups. Corresponding to the non-uniform microlens array, the display screen includes a central area, an edge area, and a transition area between the central area and the edge area; wherein: the central area of ​​the display screen is divided into four equal-area sub-blocks; the transition area of ​​the display screen is divided into three equal-area sub-blocks; and the edge area of ​​the display screen is divided into two unequal-area sub-blocks.

[0014] Furthermore, the angle and direction of the image light displayed on the display screen are modulated using a non-uniform microlens array, specifically including: controlling the microlens viewing angle spacing in the central region of the non-uniform microlens array within a first preset angle; controlling the microlens viewing angle spacing in the transition region of the non-uniform microlens array to a second preset angle; and controlling the microlens viewing angle spacing in the edge region of the non-uniform microlens array within a third preset angle. Wherein: the first preset angle is smaller than the second preset angle, and the second preset angle is smaller than the third preset angle; The angular spacing of the microlenses in the non-uniform microlens array satisfies the sparse sampling constraint condition, which is expressed as follows: ; in, Indicates the region type. Indicates the central area. Indicates the transition zone. Indicates the edge region; express The microlens arrangement density of the area for Number of viewpoints in the area Output the total number of viewpoints for the target.

[0015] Based on the aforementioned deep learning-based light field display optimization method, this invention also provides a deep learning-based light field display optimization system.

[0016] A deep learning-based light field display optimization system includes: The sparse sampling module is used to acquire multi-key-view sparse images covering multiple key views in the full target view space using a sparse sampling strategy. The deep learning module is used to build a deep learning-based viewpoint super-resolution synthesis network. The viewpoint interpolation and completion module is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image using the viewpoint super-resolution synthesis network to generate a complete image of the entire target viewpoint. The rendering and mapping module is used to synthesize the light field of the complete image from the full target view through image rendering and map it to the display screen through a macro-pixel structure. An angle-direction modulation module is used to modulate the angle and direction of the image light displayed on the display screen using a non-uniform microlens array, so that the viewer receives images from different directions at different viewing positions.

[0017] The beneficial effects of this invention are as follows: This invention provides a light field display optimization method and system based on deep learning. By employing a non-uniform microlens array to optimize the display area and combining it with deep learning algorithms to reconstruct sparse viewpoint images, it not only significantly improves the display quality and viewing angle continuity of the light field image but also enhances system performance under limited hardware resources. This invention can achieve high-quality naked-eye multi-view 3D display at low cost, possessing good practicality and scalability, and is suitable for multiple application scenarios such as augmented reality, educational displays, and medical visualization. Attached Figure Description

[0018] Figure 1 This is a flowchart of a light field display optimization method based on deep learning according to the present invention; Figure 2 This is a schematic diagram of the viewpoint super-resolution synthesis network. Figure 3 An image diagram optimized for light field display; Figure 4 This is a structural block diagram of a light field display optimization system based on deep learning according to the present invention. Detailed Implementation

[0019] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0020] like Figure 1 As shown, a deep learning-based light field display optimization method includes: S1, adopts a sparse sampling strategy to acquire multi-key-view sparse images covering multiple key views in the full target view space; S2, Construct a viewpoint super-resolution synthesis network based on deep learning; S3, the viewpoint super-resolution synthesis network is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image to generate a complete image of the entire target viewpoint. S4, the complete image of the full target view is synthesized by light field through image rendering and mapped to the display screen through macro-pixel structure; S5, using a non-uniform microlens array to modulate the angle and direction of the image light displayed on the screen, so that the viewer receives images from different directions at different viewing positions.

[0021] Specifically, non-uniform microlens arrays (MLAs) can solve the problems of smoothness and realism in visual perception caused by using uniform microlens arrays in existing technologies. By using dense microlenses in areas with high viewpoint requirements and sparse microlenses in edge areas, non-uniform MLAs optimize the viewpoint sampling method, reduce unnecessary bandwidth consumption, and improve image quality. This design allows light field displays to effectively reduce data transmission pressure while maintaining visual effects. Furthermore, the use of non-uniform MLAs improves display quality and better meets the needs of large viewpoints and high resolutions, providing a new solution for the practical application of light field displays. However, while the optimization of the optical design of non-uniform microlens arrays effectively solves some problems, it still cannot fully meet all requirements.

[0022] With the widespread application of deep learning technology, light field display technology has achieved new breakthroughs. Through deep learning networks, especially super-resolution reconstruction networks, the system can synthesize higher-resolution images from low-resolution viewpoint information, thereby further improving the display effect. This collaborative design of optics and deep learning enables light field display systems to reconstruct images more accurately, providing a more natural 3D visual experience. The combination of non-uniform microlens arrays and deep learning technologies not only solves the technical bottlenecks of light field displays but also provides an effective impetus for their widespread development in practical applications.

[0023] Therefore, this invention significantly improves the image quality of light field display by constructing a regionally designed non-uniform microlens array and combining it with a deep neural network to complete and reconstruct sparse viewpoint images, ensuring controllable system hardware costs and enhancing the naked-eye multi-viewpoint viewing experience.

[0024] The following provides a detailed explanation of each step.

[0025] In S1 above: The full target view space can be 1800 target view spaces composed of 60 horizontal directions and 30 vertical directions. Multiple key viewpoints can be 600 key viewpoints within this 1800 target view space; that is, sparse images from multiple key viewpoints cover 600 key viewpoints within the 1800 target view space composed of 60 horizontal directions and 30 vertical directions. Considering the limitations of display resolution and data transmission bandwidth, the system cannot provide independent high-resolution images for all viewpoints. Therefore, a sparse sampling strategy is adopted, acquiring only some key viewpoint images, and then using a subsequent neural network for interpolation and completion between viewpoints. The selection of the key viewpoints is determined based on the physical arrangement of the non-uniform microlens array.

[0026] In S2 above: like Figure 2 As shown, the deep learning-based viewpoint super-resolution synthesis network (VSSR-Net) includes a feature extraction module, a disparity guidance module, a global Transformer module, and a multi-scale reconstruction module; The feature extraction module employs shared-weight 3D convolutions to jointly extract spatial and viewpoint features, capturing disparity variation information to jointly model these features. The disparity guidance module, based on the disparity map predicted by the network, performs spatial distortion and weighted fusion on features from adjacent viewpoints, handling details in occlusion and boundary regions. The global Transformer module utilizes a self-attention mechanism to capture long-range dependencies between non-neighboring viewpoints, guiding structural inference in sparsely populated edge regions with a central high-density view, enhancing the consistency of the overall view. The multi-scale reconstruction module combines a residual dense network with sub-pixel convolutions, using super-resolution enhancement to upscale the input key viewpoint image (approximately 1080p) to a 2K resolution output, effectively compensating for directional blur and pixel diffusion during microlens imaging, ensuring the synthesized image retains realistic detail even after optical magnification.

[0027] Furthermore, the process of constructing the viewpoint super-resolution synthesis network mainly involves training the network. During this training, a differentiable rendering module is introduced to simulate brightness diffusion, orientation blurring, and edge distortion during the optical projection process of a non-uniform microlens array. This makes the network output more closely resemble the actual optical imaging effect, achieving end-to-end joint optimization. The differentiable rendering module is introduced during the network training phase to simulate quantization perturbations, improving the model's robustness under low bitwidth conditions and enabling real-time inference in embedded systems.

[0028] A composite loss function is used during training, and the composite loss function is expressed as follows: ; In the formula, This is a composite loss; This represents pixel error loss, reflecting the fundamental accuracy of image reconstruction. To improve texture and structure quality, we aim to detect loss. This is the epipolar consistency loss, used to constrain the geometric consistency between viewpoints; To combat the loss and enhance the realism of the synthesized image; , , , These are the loss weights corresponding to pixel error loss, perceptual loss, epipolar consistency loss, and adversarial loss, respectively.

[0029] Specifically, the training process of the viewpoint super-resolution synthesis network comprehensively introduces a composite loss function composed of multiple types of loss functions to optimize the image reconstruction results from multiple dimensions such as structural consistency, texture detail, and visual realism. The network training simultaneously considers objectives such as pixel-level error, receptive domain similarity, epipolar projection consistency, and image realism discrimination, and performs end-to-end learning through a joint loss function. These losses constrain the network in terms of basic image reconstruction accuracy, texture detail restoration, subjective realism, and multi-view geometric consistency. Specifically, the epipolar consistency loss ensures the geometric relationship of 3D point projections between stereo images, avoiding depth breaks caused by reconstruction errors; the receptive loss extracts mid-to-high frequency texture features through a pre-trained network to optimize image detail quality; and the adversarial loss uses a lightweight discriminator to guide the generated images to converge towards a realistic distribution. To improve reconstruction quality, the loss weights are dynamically adjusted according to the features of different image regions during training, with particular emphasis on optimizing occlusion boundaries and sparse texture regions to ensure stable reconstruction of image structure and details.

[0030] In addition, after the viewpoint super-resolution synthesis network is trained, it can be pruned and the model compressed to meet the deployment requirements of embedded systems.

[0031] After pruning and model compression optimization, the viewpoint super-resolution synthesis network is deployed to an image processing platform, specifically including: Statistical analysis of the APoZ metric of the convolutional channels in the viewpoint super-resolution synthesis network across the entire training dataset; Convolutional channels with APoZ values ​​greater than a preset threshold are pruned and removed, and the remaining convolutional channels are quantized using 8-bit integers to obtain a viewpoint super-resolution synthesis network optimized by pruning and quantization compression. Deploy the pruned and quantized compression optimized viewpoint super-resolution synthesis network on the image processing platform; Specifically, the APoZ metric effectively reflects channel activity and redundancy. Based on the APoZ metric, the average zero activation ratio of convolutional channels in the entire training dataset is calculated to evaluate the actual contribution of each channel in feature extraction. Channels with high APoZ metrics but low impact on the final output are pruned and removed, i.e., inefficient channels are pruned using a channel masking strategy. The retained network parameters are then quantized using 8-bit integers to reduce storage space and computational resource consumption, making them suitable for the operating efficiency and model accuracy requirements of embedded systems. This compresses model storage space and memory bandwidth, reduces dependence on 32-bit floating-point computational resources, and improves the operating efficiency of embedded devices. The specific quantization process is implemented using the following formula: ; In the formula, This is the result of 8-bit integer quantization. It has 32-bit floating-point convolution channels; It is a quantification factor, and ; It is the offset, and ;in, and These represent the maximum and minimum values ​​of the quantization weights, respectively. The quantization process uses a quantization factor and an offset to convert the weights based on their maximum and minimum values, further reducing the reliance on floating-point calculations.

[0032] Specifically, the viewpoint super-resolution synthesis network structure, after joint optimization through pruning and quantization, possesses the capability to run efficiently on embedded devices. The pruning process dynamically determines the importance of each convolutional channel based on the zero activation ratio and activation stability of statistical channels, and achieves structural-level pruning by setting channel masks. The quantization stage employs a perceptual quantization training strategy, introducing a low-bit-width simulation mechanism during training to ensure stable performance even when facing 8-bit fixed-point representations. Furthermore, to adapt to the power consumption and computing power requirements of different hardware platforms, the pruning rate and bit-width configuration are optimally combined through a joint search strategy, generating multiple model versions for dynamic switching by the system to achieve an optimal balance between accuracy, latency, and resource utilization.

[0033] In S3 above: All key viewpoint sparse images obtained in S1 are input into the viewpoint super-resolution synthesis network to reconstruct and output a complete image of the entire target viewpoint, thereby improving the resolution of each viewpoint. For example, 600 key viewpoint sparse images obtained in S1 are input into the viewpoint super-resolution synthesis network to reconstruct and output 1800 target viewpoint images, with the resolution of each viewpoint improved to 2K.

[0034] In the feature extraction module, a shared-weight 3D convolution is used to jointly extract multi-level features of the multi-key viewpoint image in both spatial and viewpoint dimensions to capture parallax changes and angular differences between viewpoints, thereby obtaining basic feature representations. In the disparity guidance module, based on the internally predicted disparity map, spatial distortion and weighted fusion are performed on the adjacent viewpoint features in the basic feature representation; In the global Transformer module, a self-attention mechanism is used to capture the long-range dependencies between non-neighboring viewpoint features in the basic feature representation; In the multi-scale reconstruction module, the outputs of the disparity guidance module and the global Transformer module are fused together, and viewpoint interpolation and completion are performed by combining a residual dense network and sub-pixel convolution, and super-resolution enhancement is performed to output a high-resolution complete image of the entire target viewpoint.

[0035] Specifically, such as Figure 2 As shown, the viewpoint super-resolution synthesis network first inputs multi-key viewpoint images into the feature extraction module. This module employs a shared-weight 3D convolutional structure to jointly extract multi-level features in both spatial and viewpoint dimensions, capturing disparity variations and angular differences between viewpoints, thereby constructing a basic feature representation with rich details and structural information. The extracted representations are then fed into the disparity guidance module and the global Transformer module to achieve deep feature fusion and spatial relationship modeling. The disparity guidance module uses disparity maps predicted within the network to spatially warp and weightedly fuse features from adjacent viewpoints, effectively mitigating information loss caused by occlusion and structural deformation, and enhancing the detail representation of local areas. Simultaneously, the global Transformer module, based on a self-attention mechanism, captures long-range dependencies between non-neighboring viewpoints and uses high-density view features in the central area to guide structural inference from sparse viewpoints in the edge areas, thereby improving the spatial consistency and structural coherence of the overall view. The fused features are then fed into the multi-scale reconstruction module, which combines a residual dense network with sub-pixel convolutional techniques to achieve progressive super-resolution reconstruction from the input level (approximately 1080p) to an output resolution of 2K. This process effectively compensates for directional blurring and pixel diffusion during microlens array imaging, ensuring the detail clarity and texture realism of the synthesized image. Simultaneously, a position-aware guidance mechanism is integrated into the multi-scale reconstruction module, ensuring that the network accurately utilizes viewpoint distribution information when processing spatial relationships from different perspectives, thereby enhancing geometric reasoning capabilities and the continuity between viewpoints.

[0036] Furthermore, in S1 above, acquiring the multi-key-view sparse image further includes: acquiring the viewpoint spatial position information corresponding to each key-view sparse image in the multi-key-view sparse image, and representing the viewpoint spatial position information as two-dimensional coordinates through a normalized mapping method, with the range mapped to the [0,1] interval, corresponding to the relative angle coordinates in the horizontal and vertical directions respectively, to obtain a two-dimensional position code. Further, in S3 above, the viewpoint super-resolution synthesis network is equipped with a position-aware guidance mechanism, which is used to assist the viewpoint super-resolution synthesis network in learning the spatial relationships between viewpoints based on the two-dimensional position code.

[0037] The position-aware guidance mechanism maps the two-dimensional normalized view coordinates (horizontal and vertical angles) of each key viewpoint sparse image to a two-dimensional position code, and uses it as an auxiliary signal in the input of the viewpoint super-resolution synthesis network to participate in the feature fusion process.

[0038] Specifically, such as Figure 2As shown, two-dimensional positional encoding is input into the viewpoint super-resolution synthesis network along with multi-key viewpoint images. The two-dimensional positional encoding is generated based on the horizontal and vertical relative angular coordinates of the acquired viewpoints, serving as an explicit guide for the network's spatial information and assisting the feature extraction module in effectively perceiving the spatial relationships between viewpoints. The local parallax processing module performs feature distortion and fusion based on the relative angles and positional encodings of adjacent viewpoints. The global Transformer module dynamically adjusts the attention coverage according to the viewpoint distribution of the input image, enabling the transfer of contextual information between non-neighboring viewpoints. This mechanism also supports embedding features from the central region as structural priors for edge regions, thereby improving the reconstruction coherence and realism of sparse regions.

[0039] In S4 above: When the complete image from the full target view is used for light field synthesis through image rendering, the image rendering employs a pixel-level aggregation function, and the optical imaging model of the pixel-level aggregation function is expressed as follows: ; In the formula, , indicating the pixel position; This indicates the final image to be displayed; Indicates perspective At pixel position Projection weights at the location; Indicates perspective At pixel position The corresponding full-target viewpoint complete image ensures that multiple viewpoint images are aggregated according to spatial structure on the display end.

[0040] Furthermore, the complete image from the entire target viewpoint is synthesized using light field synthesis and mapped onto the display screen through a macro-pixel structure to ensure optimal parallax rendering for each region. The mapping structure includes: each microlens in the non-uniform microlens array covers a corresponding macro-pixel on the display screen; the macro-pixel is internally divided into multiple sub-pixel groups for presenting viewpoint images from multiple directions. Corresponding to the non-uniform microlens array, the display screen includes a central area, an edge area, and a transition area between the central and edge areas; wherein: the central area of ​​the display screen is divided into four equal-area sub-blocks to ensure uniform output of images from multiple directions and guarantee the natural continuity of parallax in the viewing area; the transition area of ​​the display screen is divided into three equal-area sub-blocks as a buffer between the central and edge areas, smoothing parallax changes and preventing perceptual discontinuities; the edge area of ​​the display screen is divided into two unequal-area sub-blocks, with the main viewpoint sub-block accounting for more than 75% of the macro-pixel area, using brightness and sharpness enhancement strategies to compensate for information loss and reduced spatial sense caused by sparse peripheral sampling.

[0041] Specifically, the display screen is divided into three main areas: a central area, a transition area, and an edge area. Each area is configured according to viewpoint density and human visual characteristics to ensure a natural transition of parallax effects between different areas. The light field image display structure employs a macro-pixel design with regional division and a main viewing angle brightness enhancement mechanism to improve the image recognition of edge areas and overall visual coherence. Each microlens corresponds to a macro-pixel unit, which is divided into several sub-pixel groups according to the viewpoint density requirements in different areas. The central area is divided into equal-area sub-blocks to uniformly present multi-directional views; larger main viewing angle sub-blocks are set in the edge areas to enhance the visual energy output in the main observation direction. To compensate for the brightness decrease in the edge areas, the pixels in the main viewing block are adjusted through a brightness weighting function to enhance the light energy projection effect on the microlens imaging path. In addition, to improve the smoothness of viewpoint switching, the sub-blocks support asymmetrical shapes and non-uniform arrangements, making the image boundary transition more natural under multi-view observation conditions and avoiding brightness breaks and frame skipping perception caused by switching between main and auxiliary viewpoints.

[0042] This invention synthesizes the reconstructed, full-view image of the target using light field rendering and maps it onto a flat-panel display screen to achieve a spatial parallax effect. During display, a pixel-level aggregation function is used to complete the final image rendering, ensuring that multiple viewpoint images are aggregated according to spatial structure, providing a realistic 3D visual experience. The reconstruction result is mapped to the display screen through a macro-pixel structure, ensuring optimal parallax rendering for each region.

[0043] In S5 above: The configuration of the non-uniform microlens array structure is determined by a comprehensive consideration of regional viewing frequency, human visual characteristics, and attenuation models of the foveal and peripheral visual fields. The non-uniform microlens array includes a central region, an edge region, and a transition region between the central and edge regions; the configuration of the non-uniform microlens array is as follows: The central region of the non-uniform microlens array is the main viewing area, corresponding to the high-sensitivity perception area in the center of the human eye, covering a field of view of ±20°, and employing a high-density microlens arrangement to provide more viewing directions, ensuring the continuity and clarity of parallax, and each microlens projects four viewing points in different directions. The transition zone of the non-uniform microlens array covers a field of view of 20° to 30° and uses a medium-density microlens arrangement. A medium number of viewpoints are set to buffer the transition, prevent perceptual discontinuity, and ensure a smooth visual transition. Each microlens projects viewpoints in three directions. The edge region of the non-uniform microlens array is a low-sensitivity area around the human eye, covering a field of view greater than 30°. It adopts a low-density microlens arrangement, which reduces redundant data and improves system efficiency. Each microlens projects viewpoints in two directions, mainly to maintain stereoscopic contour information.

[0044] Specifically, by combining the perspective selection with the spatial arrangement density of microlenses, a gradient structure is constructed with dense microlenses in the central region and sparse microlenses in the peripheral region. This design can improve the continuity of parallax in the central region and effectively compress redundant data.

[0045] Furthermore, to meet the sensitivity requirements of the human eye to parallax changes, a non-uniform microlens array is used to modulate the angle and direction of the image light displayed on the screen. Specifically, the inter-angle spacing of the microlenses in the central region of the non-uniform microlens array is controlled within a first preset angle to ensure high parallax resolution and continuity in the fixation area, ensuring smooth depth perception and providing a natural parallax experience, especially adapting to parallax changes under slight head movements, preventing jumps in vision, and maintaining three-dimensional continuity. Considering the sensitivity of the human eye to parallax changes, the inter-angle spacing of the microlenses in the transition region of the non-uniform microlens array is controlled at a second preset angle to ensure smooth parallax changes in the transition region and avoid perceptual breaks. To ensure the continuity and natural transition of the visual experience and improve the stability of overall parallax perception, considering the low sensitivity of the human eye to edge parallax, the viewing angle spacing of the microlenses in the edge region of the non-uniform microlens array is controlled within a third preset angle. In this region, the provided viewing angle is sparse to save data acquisition and reconstruction resources. Viewpoint interpolation and completion are performed through network reconstruction technology to avoid visual discontinuity or ghosting, while ensuring viewpoint continuity and natural edge transition. Wherein, the first preset angle is smaller than the second preset angle, and the second preset angle is smaller than the third preset angle. For example, the first preset angle is set to 0.5°, the second preset angle is set to 1°, and the third preset angle is set to 2°.

[0046] The overall microlens viewing angle spacing of the non-uniform microlens array satisfies the sparse sampling constraint, meaning that the product of the microlens density and the corresponding number of projection viewpoints in different regions is much smaller than the total number of target output viewing angles of the system, thereby achieving the reconstruction of the maximum coverage viewing angle space with the least amount of sampling information. Specifically, the sparse sampling constraint is expressed as follows: ; in, Indicates the region type. Indicates the central area. Indicates the transition zone. Indicates the edge region; express The microlens arrangement density of the area for Number of viewpoints in the area Output the total number of viewpoints for the target. The sparse sampling constraint indicates that the total number of viewpoints under sparse sampling is significantly less than the target viewpoint space capacity. The region spacing and density configuration can be flexibly adjusted according to the specific display application scenario to achieve dual optimization of 3D visual effects and system efficiency.

[0047] In a non-uniform microlens array, different arrangement densities and viewing angle spacing parameters are set in the central region, transition region, and edge region to match the perceptual characteristics of the human eye in different visual fields. Each microlens determines its corresponding image projection angle based on its position coordinates in the array and the arrangement density and number of viewpoints in its region, satisfying the following mapping relationship: in, Indicates the first Projection angle of each viewpoint For region type, The coordinates of the microlenses in the array are given. and These represent the microlens density and the number of directions supported by each lens in the region, respectively. This function ensures that the spectral density and optical projection direction within each region precisely match the region's perception requirements, enabling controllable modulation of angular light rays.

[0048] This invention employs a non-uniformly arranged microlens array in front of a display screen. The array is divided into a central region, a transition region, and an edge region. The structural configuration is based on a comprehensive consideration of regional viewing frequency, human eye visual characteristics, and attenuation models of the foveal and peripheral visual fields. Each microlens can project viewpoint images from different directions according to the density and number of viewpoints in its region, satisfying the human eye's need for parallax variations. Simultaneously, the non-uniform microlens array modulates the angle and direction of image light, allowing viewers to receive images from different directions from different viewing positions. The viewing angle spacing is controlled according to the visual sensitivity of each region; a smaller viewing angle spacing is maintained in the central region to ensure smooth parallax perception, while a larger viewing angle spacing is set in the transition region to ensure a natural transition in visual experience.

[0049] Figure 3This schematic diagram illustrates an optimized imaging method for light field display. It shows that input image 1 (a sparse image with multiple key viewpoints) is processed by an embedded intelligent terminal 2 (which has a deep learning-based viewpoint super-resolution synthesis network deployed within it), output from display screen 3, and then projected onto the human eye 5 via a non-uniform microlens array 4, achieving high-definition naked-eye 3D imaging. This invention achieves complete reconstruction and multi-viewpoint display of light field images by combining the sparse sampling design of the non-uniform microlens array with the efficient image reconstruction capabilities of the deep learning network. The embedded intelligent terminal outputs a high-quality set of multi-viewpoint images with 2K resolution. These images maintain good continuity and consistency in space and viewpoint, meeting the requirements of the light field display system for the number and resolution of viewpoints. The output images serve as the basis for optical projection and light field synthesis, achieving high-fidelity visual presentation under a non-uniform microlens array.

[0050] Based on the aforementioned deep learning-based light field display optimization method, this invention also provides a deep learning-based light field display optimization system.

[0051] like Figure 4 As shown, a light field display optimization system based on deep learning includes: The sparse sampling module is used to acquire multi-key-view sparse images covering multiple key views in the full target view space using a sparse sampling strategy. The deep learning module is used to build a deep learning-based viewpoint super-resolution synthesis network. The viewpoint interpolation and completion module is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image using the viewpoint super-resolution synthesis network to generate a complete image of the entire target viewpoint. The rendering and mapping module is used to synthesize the light field of the complete image from the full target view through image rendering and map it to the display screen through a macro-pixel structure. An angle-direction modulation module is used to modulate the angle and direction of the image light displayed on the display screen using a non-uniform microlens array, so that the viewer receives images from different directions at different viewing positions.

[0052] It should be noted that the specific functions of each module in the light field display optimization system based on deep learning of the present invention are described in the specific steps of the light field display optimization method based on deep learning of the present invention, and will not be repeated here.

[0053] This invention discloses a light field display optimization method and system based on deep learning. By employing a non-uniform microlens array to optimize the display area and combining it with deep learning algorithms to reconstruct sparse viewpoint images, it not only significantly improves the display quality and viewing angle continuity of the light field image but also enhances system performance under limited hardware resources. This invention can achieve high-quality naked-eye multi-view 3D display at low cost, possessing good practicality and scalability, and is suitable for various application scenarios such as augmented reality, educational displays, and medical visualization.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A light field display optimization method based on deep learning, characterized in that, include: A sparse sampling strategy was employed to acquire multi-key-view sparse images covering multiple key viewpoints in the full target viewpoint space. Construct a deep learning-based viewpoint super-resolution synthesis network; The viewpoint super-resolution synthesis network is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image to generate a complete image of the entire target viewpoint. The complete image from the full target viewpoint is synthesized using image rendering and then mapped onto the display screen through a macro-pixel structure. The angle and direction of the image light displayed on the screen are modulated by a non-uniform microlens array so that the viewer receives images from different directions at different viewing positions.

2. The light field display optimization method based on deep learning according to claim 1, characterized in that, The non-uniform microlens array includes a central region, an edge region, and a transition region between the central region and the edge region; the non-uniform microlens array is configured as follows: The central region of the non-uniform microlens array covers a field of view of ±20° and is arranged with high density microlenses, with each microlens projecting four viewpoints in different directions. The transition zone of the non-uniform microlens array covers a field of view of 20° to 30°, and is arranged with medium density microlenses, with each microlens projecting viewpoints in three directions. The edge region of the non-uniform microlens array covers a field of view greater than 30°, and a low-density microlens arrangement is used, with each microlens projecting viewpoints in two directions.

3. The light field display optimization method based on deep learning according to claim 1, characterized in that, The viewpoint super-resolution synthesis network includes a feature extraction module, a disparity guidance module, a global Transformer module, and a multi-scale reconstruction module; The viewpoint super-resolution synthesis network is used to perform viewpoint interpolation and completion on the multi-key viewpoint images, specifically including: In the feature extraction module, a shared-weight 3D convolution is used to jointly extract multi-level features of the multi-key viewpoint image in both spatial and viewpoint dimensions to capture parallax changes and angular differences between viewpoints, thereby obtaining basic feature representations. In the disparity guidance module, based on the internally predicted disparity map, spatial distortion and weighted fusion are performed on the adjacent viewpoint features in the basic feature representation; In the global Transformer module, a self-attention mechanism is used to capture the long-range dependencies between non-neighboring viewpoint features in the basic feature representation; In the multi-scale reconstruction module, the outputs of the disparity guidance module and the global Transformer module are fused together, and viewpoint interpolation and completion are performed by combining a residual dense network and sub-pixel convolution, and super-resolution enhancement is performed to output a high-resolution complete image of the entire target viewpoint.

4. The light field display optimization method based on deep learning according to claim 1, characterized in that, The process of acquiring the multi-key-view sparse image further includes: acquiring the view spatial position information corresponding to each key-view sparse image in the multi-key-view sparse image, and representing the view spatial position information as two-dimensional coordinates through a normalized mapping method to obtain a two-dimensional position code. The viewpoint super-resolution synthesis network is equipped with a position-aware guidance mechanism, which is used to assist the viewpoint super-resolution synthesis network in learning the spatial relationships between viewpoints based on the two-dimensional position encoding.

5. The light field display optimization method based on deep learning according to claim 1, characterized in that, In constructing the viewpoint super-resolution synthesis network, a differentiable rendering module and a composite loss function are introduced to perform end-to-end joint optimization learning training on the viewpoint super-resolution synthesis network; wherein: The differentiable rendering module is used to simulate brightness diffusion, directional blurring, and edge distortion during the optical projection process of a non-uniform microlens array. The composite loss function is expressed as follows: ； In the formula, , , , , These are composite loss, pixel error loss, perceptual loss, epipolar consistency loss, and adversarial loss, respectively. , , , These are the loss weights corresponding to pixel error loss, perceptual loss, epipolar consistency loss, and adversarial loss, respectively, and the loss weights are dynamically adjusted according to the features of different image regions during training.

6. The light field display optimization method based on deep learning according to claim 1, characterized in that, Before using the viewpoint super-resolution synthesis network to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image, the method further includes: pruning and model compression optimization of the viewpoint super-resolution synthesis network and then deploying it to the image processing platform. Specifically, the viewpoint super-resolution synthesis network is pruned and optimized using model compression before being deployed to an image processing platform, including: Statistical analysis of the APoZ metric of the convolutional channels in the viewpoint super-resolution synthesis network across the entire training dataset; Convolutional channels with APoZ values ​​greater than a preset threshold are pruned and removed, and the remaining convolutional channels are quantized using 8-bit integers to obtain a viewpoint super-resolution synthesis network optimized by pruning and quantization compression. Deploy the pruned and quantized compression optimized viewpoint super-resolution synthesis network on the image processing platform; The quantization process is implemented using the following formula: ； In the formula, This is the result of 8-bit integer quantization. It has 32-bit floating-point convolution channels; As a quantification factor, and ; It is the offset, and ;in, and These are the maximum and minimum values ​​of the quantization weights, respectively.

7. The light field display optimization method based on deep learning according to claim 1, characterized in that, When the complete image from the full target view is used for light field synthesis through image rendering, the image rendering employs a pixel-level aggregation function, and the optical imaging model of the pixel-level aggregation function is expressed as follows: ； In the formula, , indicating the pixel position; This indicates the final image to be displayed; Indicates perspective At pixel position Projection weights at the location; Indicates perspective At pixel position The corresponding complete image of the entire target from all angles.

8. The light field display optimization method based on deep learning according to claim 2, characterized in that, When the complete image from the full target view is synthesized using light field and mapped onto the display screen through a macro-pixel structure, the mapping structure includes: Each microlens in the non-uniform microlens array covers a corresponding macropixel in the display screen, and the macropixel is divided into multiple sub-pixel groups. Corresponding to the non-uniform microlens array, the display screen includes a central area, an edge area, and a transition area between the central area and the edge area; wherein: the central area of ​​the display screen is divided into four equal-area sub-blocks; the transition area of ​​the display screen is divided into three equal-area sub-blocks; and the edge area of ​​the display screen is divided into two unequal-area sub-blocks.

9. The light field display optimization method based on deep learning according to claim 2, characterized in that, The method of using a non-uniform microlens array to modulate the angle and direction of the image light displayed on the display screen specifically includes: controlling the viewing angle spacing of the microlenses in the central region of the non-uniform microlens array within a first preset angle; controlling the viewing angle spacing of the microlenses in the transition region of the non-uniform microlens array to a second preset angle; and controlling the viewing angle spacing of the microlenses in the edge region of the non-uniform microlens array within a third preset angle. Wherein: the first preset angle is smaller than the second preset angle, and the second preset angle is smaller than the third preset angle; The angular spacing of the microlenses in the non-uniform microlens array satisfies the sparse sampling constraint condition, which is expressed as follows: ； in, Indicates the region type. Indicates the central area. Indicates the transition zone. Indicates the edge region; express The microlens arrangement density of the area for Number of viewpoints in the area Output the total number of viewpoints for the target.

10. A light field display optimization system based on deep learning, characterized in that, include: The sparse sampling module is used to acquire multi-key-view sparse images covering multiple key views in the full target view space using a sparse sampling strategy. The deep learning module is used to build a deep learning-based viewpoint super-resolution synthesis network. The viewpoint interpolation and completion module is used to perform viewpoint interpolation and completion on the multi-key viewpoint sparse image using the viewpoint super-resolution synthesis network to generate a complete image of the entire target viewpoint. The rendering and mapping module is used to synthesize the light field of the complete image from the full target view through image rendering and map it to the display screen through a macro-pixel structure. An angle-direction modulation module is used to modulate the angle and direction of the image light displayed on the display screen using a non-uniform microlens array, so that the viewer receives images from different directions at different viewing positions.

Images