A light field display adaptive rendering method and system based on user pose perception

Abstract

The present application relates to the technical field of light field display and image generation, and particularly relates to a light field display adaptive rendering method and system based on user pose perception. The method acquires the head pose of a user, calculates the three-dimensional position of the user in the coordinate system of a light field terminal; calculates the target viewing angle accordingly, and calculates the priority value of the viewing angle based on the angular distance between each discrete display viewing angle and the target viewing angle; divides each viewing angle into a focal point area, a transition area and a peripheral area according to the priority, and constructs high and low priority viewing angle sets; performs full-quality independent rendering based on a three-dimensional scene for the viewing angles in the high priority set; performs reduced-quality independent rendering for the viewing angles in the low priority set, or generates viewing angle images by using image deformation technology with reference to a color map and a depth map; and finally combines and outputs the rendered images. The present application establishes a differentiated generation mechanism across viewing angles, greatly reduces the computing power load of light field display, and ensures stable main viewing angle image quality and output frame rate.

Description

Technical Field

[0001] This invention relates to the field of light field display and image generation technology, and in particular to a light field display adaptive rendering method and system based on user pose awareness. Background Technology

[0002] Light field display devices emit light rays carrying corresponding viewpoint information in different spatial directions, allowing users to perceive a three-dimensional effect without wearing auxiliary equipment. To achieve this effect, the light field display panel needs to output images from multiple discrete viewpoints simultaneously, with the number of viewpoints typically ranging from 8 to 45 depending on the panel specifications. In existing multi-view rendering solutions, if a complete rendering based on the 3D scene is performed for each discrete viewpoint, the overall system rendering load will increase approximately linearly with the number of viewpoints. On consumer-grade terminals with limited computing power, this uniform quality global rendering leads to severe frame rate drops, screen stuttering, and high latency, failing to meet the requirements for smooth interaction.

[0003] Existing multi-view rendering methods for light field displays are mostly based on a globally equal-quality rendering architecture or borrow from the single-image foveated rendering model in the field of virtual reality. They primarily reduce GPU computing power by lowering the 2D image resolution in the area surrounding the focal point. In the research and application of multi-view light field display devices, traditional technologies typically treat the dozens of discrete viewpoints output to the screen as independent rendering task pools, or simply use a basic gaze-tracking algorithm for coarse-grained global degradation to alleviate the load on the terminal hardware. These methods often rely on spatial degradation theory within a single frame image for data processing, allocating the proportion of computational resources within a single viewpoint based on the empirical law that human visual acuity monotonically decreases with the center angle.

[0004] However, the aforementioned existing technologies have significant drawbacks when handling spatial array rendering with massive discrete viewpoints, such as light field displays. First, traditional methods struggle to accurately depict the cross-modal mapping relationship between the user's 3D physical pose and multiple viewpoints. In particular, single-viewpoint rendering is limited to pixel-level frequency reduction within the 2D field of view plane, and its static allocation strategy cannot reflect the dynamic differences in viewpoint importance as the 3D spatial angular distance between the "target viewing angle" and "each discrete hardware viewpoint" changes. Second, existing multi-view rendering technologies lack computational environment sensitivity adjustment logic, ignoring the "squeezing" effect of concurrent tasks on the terminal system and real-time GPU load fluctuations on the rendering pipeline. Fixed rendering configurations make it easy for the same device to experience frame rate drops and screen stuttering under different complex scenes or pose changes. Finally, fixed rendering modes or simple global degradation models often deviate from user visual perception and 3D geometric parallax mechanisms, failing to distinguish the importance differences of each viewpoint based on the user's current spatial position. This easily results in either severe power overload or severe global image quality loss, making it difficult to solve the practical application challenge of ensuring the ultimate image quality of the main viewpoint while drastically reducing the overall rendering cost on computing-constrained terminals.

[0005] The applicant conducted a targeted search of existing literature in the fields of light field display and multi-view display. While existing light field display terminals employ schemes that adjust the display viewpoint selection or viewpoint order by tracking the user, their purpose is to select which pre-generated viewpoint content to present to the user, rather than differentiating the image generation methods for each viewpoint based on the user's spatial pose during real-time rendering. Some literature uses depth-based image deformation for view synthesis in multi-view video systems, but this operates at the content encoding or transmission link level and does not address the issue of dynamically dividing viewpoint sets based on the user's real-time pose and performing independent rendering and image deformation on different sets in online rendering scenarios. Overall, in real-time rendering scenarios for computationally limited light field display terminals, no publicly available technical solutions have been found that correlate the user's 3D spatial pose in the panel coordinate system with the image generation strategies of each discrete viewpoint and achieve differentiated allocation of viewpoint-level rendering resources based on an angular distance priority function. Summary of the Invention

[0006] To address the technical problems of existing light field display multi-view rendering methods, which struggle to accurately construct cross-modal mapping relationships between user pose and multiple viewpoints, lack reasonable allocation of rendering computing power, and are prone to quality loss when using fixed rendering modes or simple global degradation models, resulting in poor user-centric viewpoint rendering quality, this invention provides a light field display adaptive rendering method and system based on user pose perception.

[0007] One such method for adaptive rendering of light field display based on user pose awareness includes the following steps: S1. Acquire user head pose data through multi-source data acquisition, and calculate the three-dimensional position coordinates of user head pose data in the coordinate system of the light field display terminal; S2. Calculate the target viewing angle of the user relative to the light field display terminal based on the three-dimensional position coordinates, and calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle; S3. Based on the obtained priority values ​​and combined with the pre-set priority thresholds, each discrete display viewpoint is divided into a focus area, a transition area, and an outer area. A high-priority viewpoint set is constructed using the focus area, and a low-priority viewpoint set is constructed using the transition area and the outer area. S4. Based on the pre-set target resolution, perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene to obtain the first rendering image set. Perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene to obtain the second rendering image set. Based on the first rendering image set and the second rendering image set, generate the third rendering image set by image deformation for each discrete display view in the outer area of ​​the low-priority view set. S5. Based on the obtained first, second, and third rendered image sets, combine them according to a preset viewpoint format and output the final rendered image.

[0008] Furthermore, step S1 includes the following sub-steps: S101. Acquire user images through an image acquisition device deployed within the light field display terminal, and extract user feature information, wherein the user feature information includes facial key point information; S102. Based on user feature information and combined with a preset reference model, solve the pose parameters of the user's head relative to the image acquisition device to obtain user head pose data; S103. Based on the user's head pose data, the user's head pose data is mapped into three-dimensional position coordinates of the user's head in the coordinate system of the light field display terminal through the spatial transformation relationship between the pre-calibrated coordinate system of the image acquisition device and the coordinate system of the light field display panel, and then time-series smoothing is performed through Kalman filtering or extended Kalman filtering.

[0009] Furthermore, step S102 includes the following sub-steps: S1021. Obtain a three-dimensional face reference model as a reference model, and based on the correspondence between the three-dimensional coordinates of each key point in the three-dimensional face reference model and the two-dimensional coordinates of the corresponding key points in the user feature information, use the perspective n-point algorithm to solve the rotation parameters and translation parameters. S1022. Determine the three-dimensional coordinates of the midpoint of the user's eyes in the coordinate system of the image acquisition device based on the rotation and translation parameters, and use them as the position coordinates of the user's head in the coordinate system of the image acquisition device.

[0010] Furthermore, step S2 includes the following sub-steps: S201: Calculate the target viewing angle of the user's head relative to the light field display terminal based on the three-dimensional position coordinates obtained in step S1; S202: Based on the target viewing angle, calculate the angular distance between the angle corresponding to each discrete display viewing angle of the light field display terminal and the angle corresponding to the target viewing angle. Substitute the angular distance as an independent variable into the priority function to generate the priority value for each discrete display viewing angle. The priority function is expressed as follows: ; in, This represents the priority value for each discrete display viewpoint. Indicates the first Each discrete display viewing angle corresponds to a specific angle. Indicates the target viewing angle. Indicates the attenuation parameter. This represents an exponential function.

[0011] Furthermore, step S3 includes the following sub-steps: S301. Based on the actual working parameters of the light field display terminal, set high-priority threshold and low-priority threshold; S302. Mark the viewpoints with priority values ​​not lower than the high priority threshold as the focus area, and construct a high priority viewpoint set with all discrete display viewpoints within the focus area; S303. Views with priority values ​​lower than the high priority threshold but not lower than the low priority threshold are marked as transition zones, and views with priority values ​​lower than the low threshold are marked as peripheral zones. A set of low priority views is constructed using all discrete display views within the transition zone and peripheral zone.

[0012] Furthermore, step S4 includes the following sub-steps: S401. Obtain the high-priority view set, the low-priority view image set, and the preset rendering target resolution; S402. Based on the high-priority view set, and taking the rendering target resolution as a benchmark, combine the complete lighting calculation pipeline and perform full-quality independent 3D rendering on each discrete display view within the high-priority view set to obtain the first rendering image set. S403. Based on the low-priority view set, perform degraded independent rendering on each discrete display view marked as the transition zone within the low-priority view set to obtain a second rendering image set. The degraded independent rendering includes at least rendering at a resolution lower than the rendering target resolution and then upsampling to the rendering target resolution, reducing the geometric detail level, turning off real-time shadow calculation, turning off ambient occlusion, or reducing the number of illumination samplings. S404. Based on the low-priority viewpoint set, each discrete display viewpoint marked as the peripheral area within the low-priority viewpoint set is treated as a part that does not perform independent 3D rendering. Using the color map and depth map of the already generated rendered images in the first and second rendered image sets as references, a third rendered image set is generated through depth-based image deformation. The depth-based image deformation is represented as follows: ; in, Represents pixels The horizontal displacement relative to the reference viewpoint image in the target viewpoint image. This represents the equivalent focal length parameter of the light field display terminal. This represents the baseline distance between the target viewpoint and the reference viewpoint. Represents pixels in the reference view depth map The depth value at that location.

[0013] Furthermore, the following steps are included before performing step S4: A1. Real-time acquisition of available computing power for image rendering in the current frame, dynamic adjustment of the division boundary between the high-priority view set and the low-priority view set and / or the rendering quality level of each view, so that the total cost of image generation for all views in the current frame does not exceed the preset computing power budget. A2. When the real-time available computing power of the system decreases, reduce the rendering quality level of the viewpoints in the low-priority viewpoint set. If the constraints are still not met, narrow the range of the high-priority viewpoint set, and mark the discrete display viewpoint that is closest to the target observation angle as the optimal viewpoint, and always keep it rendered independently.

[0014] Furthermore, it also includes extrapolating and predicting the user's observation point position within a future preset time window based on the user's historical pose sequence, and calculating the predicted target observation angle; when the optimal viewpoint corresponding to the predicted target observation angle is different from the current optimal viewpoint, pre-rendering is performed on at least some high-priority viewpoints corresponding to the predicted target observation angle during the rendering idle period of the current frame and the results are written to the cache; when the angle deviation between the actual user pose and the predicted pose in the next frame exceeds a preset deviation threshold, a temporary image generated by image deformation based on the current existing reference viewpoint is output first, and the temporary image is replaced by an independent rendering result in subsequent frames.

[0015] Furthermore, when the image acquisition device fails to detect valid user feature information within a preset number of consecutive frames, it continues to generate images using the viewpoint segmentation scheme from the most recent valid detection. When the duration of the failure to detect valid user feature information exceeds a preset timeout threshold, it switches to uniformly degraded rendering for all discrete display viewpoints. Hysteresis control is performed on the segmentation boundary between consecutive frames. When a viewpoint with a priority value near the segmentation threshold undergoes a change in its set affiliation, its priority value is required to remain within the threshold range corresponding to the new set for at least a preset number of consecutive frames before set switching can be performed.

[0016] This invention also provides a user pose-aware adaptive rendering system for light field display, which is implemented based on any of the user pose-aware adaptive rendering methods for light field display described above, and includes: The image acquisition and pose conversion module is used to acquire user head pose data through multi-source data acquisition and calculate the three-dimensional position coordinates of the user head pose data in the coordinate system of the light field display terminal. The priority calculation module is used to calculate the user's target viewing angle relative to the light field display terminal based on the three-dimensional position coordinates, and to calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle. The viewpoint segmentation module is used to divide each discrete display viewpoint into a focus area, a transition area, and an outer area based on the obtained priority value and a pre-set priority threshold, and to construct a high-priority viewpoint set with the focus area and a low-priority viewpoint set with the transition area and the outer area. The image rendering generation module is used to perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene, with a pre-set rendering target resolution as the benchmark, to obtain a first rendering image set; to perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene, to obtain a second rendering image set; and to generate a third rendering image set based on the first and second rendering image sets and the discrete display view in the outer area of ​​the low-priority view set through image deformation. The image output module is used to combine the obtained first, second, and third rendered image sets according to a preset viewpoint arrangement format and output the final rendered image.

[0017] A computer-readable storage medium for storing a computer program that, when run on a computer, causes the computer to perform any of the above-mentioned user pose-aware adaptive rendering methods for light field display.

[0018] A terminal device includes a light field display panel for outputting rendered images; an image acquisition device for acquiring user images; a memory for storing computer programs; and a processor for executing the computer programs to implement a user pose-aware adaptive rendering method for light field display as described above.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention abandons traditional single-image foveated rendering. By introducing a three-level spatial partitioning model of focal area, transition area, and peripheral area, and a viewpoint-level priority evaluation mechanism based on angular distance, it matches three heterogeneous pipelines: full-quality rendering, degraded rendering, and depth-based image deformation. This concentrates GPU computing power on the focal area that the user is actually observing, enabling computing power allocation to accurately follow the user's three-dimensional spatial displacement, significantly reducing the rendering cost of non-observation areas. At the same time, it adopts smoothing filtering and hysteresis control mechanisms to avoid frequent switching of viewpoint sets caused by slight head tremors of the user, ensuring the continuity of the image presentation. On the other hand, combined with a dynamic degradation and boundary adjustment mechanism based on real-time available computing power monitoring, as well as extrapolation prediction and pre-rendering strategies, it ensures that the system can maintain a stable frame rate output even under extreme conditions such as computing power peaks or tracking loss, avoiding screen stuttering. Attached Figure Description

[0020] Figure 1 This is a flowchart of a light field display adaptive rendering method based on user pose perception proposed in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of a user pose-aware light field display adaptive rendering system proposed in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the terminal device structure of a user pose-aware light field display adaptive rendering method proposed in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of a computer-readable storage medium structure for an adaptive rendering method for light field display based on user pose awareness, as proposed in an embodiment of the present invention.

[0024] In the diagram, 200 is the terminal device, 210 is the memory, 211 is the RAM, 212 is the cache memory, 213 is the ROM, 214 is the program / utility, 215 is the program module, 220 is the processor, 230 is the bus, 240 is the external device, 250 is the I / O interface, 260 is the network adapter, and 300 is the program product. Detailed Implementation

[0025] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, and not all of them. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0028] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or machine that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or machine. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or machine that includes said element.

[0029] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0030] Example 1: like Figure 1 As shown, this embodiment provides a user pose-aware adaptive rendering method for light field display, including the following steps: S1. Acquire user head pose data through multi-source data acquisition, and calculate the three-dimensional position coordinates of user head pose data in the coordinate system of the light field display terminal; Furthermore, step S1 includes the following sub-steps: S101. Acquire user images through an image acquisition device deployed within the light field display terminal, and extract user feature information, wherein the user feature information includes facial key point information; S102. Based on user feature information and combined with a preset reference model, solve the pose parameters of the user's head relative to the image acquisition device to obtain user head pose data; S103. Based on the user's head pose data, the user's head pose data is mapped into three-dimensional position coordinates of the user's head in the coordinate system of the light field display terminal through the spatial transformation relationship between the pre-calibrated coordinate system of the image acquisition device and the coordinate system of the light field display panel, and then time-series smoothing is performed through Kalman filtering or extended Kalman filtering.

[0031] Furthermore, step S102 includes the following sub-steps: S1021. Obtain a three-dimensional face reference model as a reference model, and based on the correspondence between the three-dimensional coordinates of each key point in the three-dimensional face reference model and the two-dimensional coordinates of the corresponding key points in the user feature information, use the perspective n-point algorithm to solve the rotation parameters and translation parameters. S1022. Determine the three-dimensional coordinates of the midpoint of the user's eyes in the coordinate system of the image acquisition device based on the rotation and translation parameters, and use them as the position coordinates of the user's head in the coordinate system of the image acquisition device.

[0032] Specifically, this step focuses on user viewpoint tracking in a light field display system. It acquires the user's real-time state in the physical world through visual perception in front of the target terminal, and maps the user's absolute spatial coordinates relative to the physical luminescent panel using rigid body matrix transformation and filtering algorithms. Existing technologies for eye or head tracking often rely on calculations within a single, linear camera plane coordinate system. The data output is isolated from the screen's physical coordinates, resulting in a lack of absolute spatial reference for subsequent rendering frustum calculations. This method, by introducing a 3D reference model and cross-coordinate system matrix mapping, breaks down the spatial silos between the sensor and the display panel, constructing a high-fidelity underlying spatial state dataset for subsequent cross-viewpoint addressing.

[0033] Specifically, the implementation principle described above is as follows: First, a multi-source image acquisition device deployed within the light field display terminal extracts the user's facial contours and key feature points in parallel at high frequency. The acquired two-dimensional feature information is then matched with a pre-set three-dimensional face reference model to deduce the user's six-degree-of-freedom pose of the head in sensor space. Furthermore, it should be noted that the stereoscopic vision generation of the light field display is limited by the relative emission angle between the user's eyes and the physical pixel array of the panel. Spatial transformation is achieved through a pre-fixed device calibration matrix, and a Kalman filter algorithm is superimposed to filter out high-frequency noise. This overcomes the drawback of traditional single-frame image recognition being susceptible to ambient light jitter, converging discrete physical quantities reflecting the user's observation angle into a smooth, continuous, and strongly temporally correlated spatial coordinate stream, providing a high-precision coordinate foundation for subsequent target observation angle calculations.

[0034] S2. Calculate the target viewing angle of the user relative to the light field display terminal based on the three-dimensional position coordinates, and calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle; Furthermore, step S2 includes the following sub-steps: S201: Calculate the target viewing angle of the user's head relative to the light field display terminal based on the three-dimensional position coordinates obtained in step S1; S202: Based on the target viewing angle, calculate the angular distance between the angle corresponding to each discrete display viewing angle of the light field display terminal and the angle corresponding to the target viewing angle. Substitute the angular distance as an independent variable into the priority function to generate the priority value for each discrete display viewing angle. The priority function is expressed as follows: ; in, This represents the priority value for each discrete display viewpoint. Indicates the first Each discrete display viewing angle corresponds to a specific angle. Indicates the target viewing angle. Indicates the attenuation parameter. This represents an exponential function.

[0035] Specifically, this step addresses the allocation of computational resources for multi-viewpoint light field displays. It mathematically quantifies the geometric angle between the physical projection direction of discrete viewpoints and the user's line of sight, constructing a Gaussian function priority distribution model based on angular distance. Existing technologies, when performing rendering degradation, largely rely on the gaze point mapping theory within a single VR planar image. Their resource attenuation mechanisms are limited to the 2D pixel plane and fail to address the spatial dimension of multi-viewpoint collaboration in light fields. This method, by establishing a distance response function in the angular domain, transforms the nonlinear distribution of the user's viewing direction into a quantifiable weight scalar. This overcomes the shortcomings of traditional methods in characterizing cross-viewpoint computational sensitivity, enabling the conversion of rendering resources from planar two-dimensional partitioning to a spatial three-dimensional perspective.

[0036] Specifically, the implementation principle described above is as follows: First, trigonometric functions are used to map the acquired user's 3D position system onto the normal vector domain of the terminal's emitting plane, calculating the core axis that dominates visual perception at the current moment, i.e., the target viewing angle. Simultaneously, the discrete emission angles inherent in the light field display panel hardware are sequentially subtracted from the target viewing angle, and this angular distance is input as an independent variable into a preset Gaussian attenuation model. Furthermore, it should be noted that the perceived quality of 3D stereoscopic images by the human visual system depends on the specific viewing angle falling within the pupil's range. Through the nonlinear constraints of the Gaussian distribution, the visual shift of the human eye to a viewpoint deviating from the primary viewing area is reconstructed, providing the underlying decision-making logic for subsequent rendering pipeline segmentation.

[0037] S3. Based on the obtained priority values ​​and combined with the pre-set priority thresholds, each discrete display viewpoint is divided into a focus area, a transition area, and an outer area. A high-priority viewpoint set is constructed using the focus area, and a low-priority viewpoint set is constructed using the transition area and the outer area. Furthermore, step S3 includes the following sub-steps: S301. Based on the actual working parameters of the light field display terminal, set high-priority threshold and low-priority threshold; S302. Mark the viewpoints with priority values ​​not lower than the high priority threshold as the focus area, and construct a high priority viewpoint set with all discrete display viewpoints within the focus area; S303. Views with priority values ​​lower than the high priority threshold but not lower than the low priority threshold are marked as transition zones, and views with priority values ​​lower than the low threshold are marked as peripheral zones. A set of low priority views is constructed using all discrete display views within the transition zone and peripheral zone.

[0038] Specifically, this step involves hierarchical classification of each discrete viewpoint within a multi-view matrix. Existing technologies, limited by computational resources, often employ a one-size-fits-all approach to globally degraded rendering, leading to an overall collapse in the subjective resolution of the entire light field image. This method, by establishing a hierarchical structure of focal area, transition area, and peripheral area, effectively distinguishes core display data from visually redundant data, avoiding the overflow of computational resources in useless directions. Furthermore, the buffer zone design in the transition area eliminates abrupt image changes that might be caused by abrupt segmentation.

[0039] Specifically, the implementation principle described above is as follows: First, the system reads the current performance boundary pool, anchors the threshold baselines for both high and low dimensions, and performs discrete segmentation on the continuous priority distribution field output from step S2: high-weighted views with priority values ​​residing at the top are assigned to the focus area, while views in the middle and bottom are sequentially assigned to the transition and peripheral areas. Furthermore, it should be noted that the three-dimensional continuity of the light field display is highly susceptible to causing dizziness due to the absence of individual views. By establishing a three-tiered architecture of "high and low priority sets," the system can transform the large and disordered multi-view rendering task into a structured and controllable subset of tasks at the underlying scheduler level. This allows secondary views to be input into the low-power generation pipeline while maintaining clarity of the core views.

[0040] S4. Based on the pre-set target resolution, perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene to obtain the first rendering image set. Perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene to obtain the second rendering image set. Based on the first rendering image set and the second rendering image set, generate the third rendering image set by image deformation for each discrete display view in the outer area of ​​the low-priority view set. Furthermore, step S4 includes the following sub-steps: S401. Obtain the high-priority view set, the low-priority view image set, and the preset rendering target resolution; S402. Based on the high-priority view set, and taking the rendering target resolution as a benchmark, combine the complete lighting calculation pipeline and perform full-quality independent 3D rendering on each discrete display view within the high-priority view set to obtain the first rendering image set. S403. Based on the low-priority view set, perform degraded independent rendering on each discrete display view marked as the transition zone within the low-priority view set to obtain a second rendering image set. The degraded independent rendering includes at least rendering at a resolution lower than the rendering target resolution and then upsampling to the rendering target resolution, reducing the geometric detail level, turning off real-time shadow calculation, turning off ambient occlusion, or reducing the number of illumination samplings. S404. Based on the low-priority viewpoint set, each discrete display viewpoint marked as the peripheral area within the low-priority viewpoint set is treated as a part that does not perform independent 3D rendering. Using the color map and depth map of the already generated rendered images in the first and second rendered image sets as references, a third rendered image set is generated through depth-based image deformation. The depth-based image deformation is represented as follows: ; in, Represents pixels The horizontal displacement relative to the reference viewpoint image in the target viewpoint image. This represents the equivalent focal length parameter of the light field display terminal. This represents the baseline distance between the target viewpoint and the reference viewpoint. Represents pixels in the reference view depth map The depth value at that location.

[0041] Specifically, this step addresses the differences in visual sensitivity across different regions by introducing a cross-modal hybrid generation model that couples the GPU rendering pipeline with a depth-based image deformation algorithm. Existing technologies, even with dynamic frame rates or global resolution reduction, still suffer from the high costs of 3D ray tracing and rasterization calculations when dealing with large-scale multi-view displays, resulting in a theoretical bottleneck in computational power reduction. This method completely breaks through the traditional mindset that all displays require rendering. It directly removes the 3D rendering load from the peripheral areas, using geometric deduction of optical parallax to replace complex graphics calculations, achieving a qualitative leap in the efficiency of light field image generation.

[0042] Specifically, the implementation principle described above is as follows: First, for high-priority focal areas, the system maximizes peak computing power across the entire lighting pipeline, outputting a first set of rendered images with absolute precision and physical depth. For transition areas, lightweight calculations are performed by truncating secondary rendering branches, outputting a second set of rendered images. Most importantly, for the peripheral areas, the system extracts and caches the color and depth maps already rendered in the focal or transition areas. Based on depth, image deformation directly deduces the stereoscopic image from the peripheral perspective through the lateral displacement of two-dimensional pixels. Furthermore, it should be noted that due to the strong visual overlap between different viewpoints in a 3D scene, the depth-guided image deformation algorithm extracts the internal color and depth prior features of the rendered viewpoints, significantly reducing geometric redundancy between spatial views. While deceiving peripheral vision, it avoids numerous complex 3D shading operations, effectively optimizing the allocation of rendering computing power.

[0043] S5. Based on the obtained first, second, and third rendered image sets, combine them according to a preset viewpoint format and output the final rendered image.

[0044] Furthermore, the following steps are included before performing step S4: A1. Real-time acquisition of available computing power for image rendering in the current frame, dynamic adjustment of the division boundary between the high-priority view set and the low-priority view set and / or the rendering quality level of each view, so that the total cost of image generation for all views in the current frame does not exceed the preset computing power budget. A2. When the real-time available computing power of the system decreases, reduce the rendering quality level of the viewpoints in the low-priority viewpoint set. If the constraints are still not met, narrow the range of the high-priority viewpoint set, and mark the discrete display viewpoint that is closest to the target observation angle as the optimal viewpoint, and always keep it rendered independently.

[0045] Specifically, this step addresses the issue of computing power allocation under dynamic load conditions in light field display terminals by introducing a closed-loop computing power budget control mechanism at the front end of the rendering pipeline to dynamically adjust the rendering resource configuration for each viewpoint. Existing technologies often employ fixed rendering pipeline configurations for multi-view 3D rendering. When system concurrency increases or scene complexity rises, these fixed rendering parameters can exceed the system's computing power limit, leading to frame rate drops or rendering interruptions. This method, through the dynamic coupling of real-time computing power monitoring and rendering quality, breaks the fixed computing load pattern in traditional rendering architectures, constructing an adaptive computing power allocation dataset for stable frame rate output under fluctuating computing power conditions.

[0046] Specifically, the implementation principle described above is as follows: First, the monitoring unit deployed in the rendering module obtains the available GPU time slice for the current frame in real time as the computing power budget, and calculates the total rendering cost under the current partition configuration. When the total rendering cost exceeds the computing power budget, degradation instructions are triggered in a preset order: the rendering quality parameters of low-priority sets are reduced first; if the budget is still exceeded, the partition threshold of high-priority sets is increased to reduce the number of high-priority viewpoints, while maintaining the independent 3D rendering configuration of the optimal viewpoint unchanged. Furthermore, it should be noted that the output frame rate of the light field display system is limited by the dynamically changing real-time GPU load. The dynamic adjustment of the rendering configuration is achieved through a computing power closed-loop control network, breaking the global frame drop phenomenon caused by computing power overload under traditional fixed parameters. This constrains the image quality degradation caused by insufficient computing power to the low-priority viewpoint region, providing computational mechanism support for continuous frame output under resource-constrained conditions.

[0047] Furthermore, it also includes extrapolating and predicting the user's observation point position within a future preset time window based on the user's historical pose sequence, and calculating the predicted target observation angle; when the optimal viewpoint corresponding to the predicted target observation angle is different from the current optimal viewpoint, pre-rendering is performed on at least some high-priority viewpoints corresponding to the predicted target observation angle during the rendering idle period of the current frame and the results are written to the cache; when the angle deviation between the actual user pose and the predicted pose in the next frame exceeds a preset deviation threshold, a temporary image generated by image deformation based on the current existing reference viewpoint is output first, and the temporary image is replaced by an independent rendering result in subsequent frames.

[0048] Specifically, this step addresses the rendering system response latency issue caused by dynamic user pose changes by introducing a prediction-based pre-rendering mechanism based on historical poses and an image compensation mechanism for deviations exceeding limits. Existing technologies often employ a serial processing chain when handling user tracking and image rendering, with data acquisition, calculation, and rendering occurring sequentially, leading to image response lag when user poses change rapidly. This method, through temporal extrapolation prediction and asynchronous rendering during idle periods, combined with 2D image deformation compensation technology, breaks the limitations of traditional 3D rendering pipelines' serial waiting, constructing an image dataset with temporal compensation capabilities to reduce end-to-end display latency.

[0049] Specifically, the implementation principle described above is as follows: First, by extracting the user pose coordinate sequence of consecutive frames and combining it with a kinematic model, the predicted target viewing angle after a preset time window is calculated. When the system determines that the predicted optimal viewpoint has shifted, it calls the idle time slice after the current frame's 3D rendering is completed, performs pre-rendering on the focal area corresponding to the predicted target viewpoint, and stores the data in a cache. When the deviation between the actual target pose and the predicted pose in the next frame exceeds a threshold, the system directly calls the reference depth map to perform 2D pixel remapping to output a temporary image, and simultaneously performs independent 3D rendering of that viewpoint in the background. Furthermore, it should be noted that the response latency of the display system is limited by the fixed time consumption of 3D rasterization rendering. By using prediction extrapolation and pre-rendering to achieve advance processing in timing, the system breaks the lag feedback of traditional pipelines during pose changes, replacing the time-consuming 3D rendering process with low-latency 2D image deformation as transient compensation, providing a data compensation base for the system's low-latency visual interaction.

[0050] Furthermore, when the image acquisition device fails to detect valid user feature information within a preset number of consecutive frames, it continues to generate images using the viewpoint segmentation scheme from the most recent valid detection. When the duration of the failure to detect valid user feature information exceeds a preset timeout threshold, it switches to uniformly degraded rendering for all discrete display viewpoints. Hysteresis control is performed on the segmentation boundary between consecutive frames. When a viewpoint with a priority value near the segmentation threshold undergoes a change in its set affiliation, its priority value is required to remain within the threshold range corresponding to the new set for at least a preset number of consecutive frames before set switching can be performed.

[0051] Specifically, this step addresses the issues of sensor data loss and threshold-critical state oscillations by introducing state preservation, timeout switching mechanisms, and hysteresis control logic at the viewpoint boundary into the system control layer. Existing technologies often interrupt rendering or reset the system when faced with sensor tracking loss. Furthermore, when priority determination is at the threshold edge, even small data fluctuations can lead to frequent switching of the viewpoint rendering mode, causing screen flickering. This method, by setting a time threshold and a state preservation integrator, breaks the system oscillations caused by the direct rigid mapping between the underlying sensor input state and the top-level rendering configuration, thus constructing a fault-tolerant control dataset for stable system operation under unstable input conditions.

[0052] Specifically, the implementation principle described above is as follows: First, in control flow data processing, when an interruption of the valid user feature flow is detected, the system latches the viewpoint partition parameter set of the previous valid frame to continue maintaining the rendering instructions. If the cumulative interruption time exceeds the timeout threshold, a rendering instruction to reduce the uniform resolution for all views is issued to the rendering module. Simultaneously, during viewpoint partitioning, if the priority value of a viewpoint crosses the partitioning threshold, the system starts a frame counter. Only when the priority value remains within the new interval threshold for multiple consecutive frames is the viewpoint's set assignment updated. Furthermore, it should be noted that the system's operational stability is limited by physical environment interference and sensor detection accuracy. By using state latching and timeout degradation mechanisms to handle sensor blind spots, the system overcomes rendering pipeline interruptions caused by data loss, transforming minor data fluctuations at boundaries into stable and continuous control signal outputs. This provides a fault-tolerant logical operation base for light field display systems in complex physical environments.

[0053] Example 2

[0054] like Figure 2 As shown, as a preferred embodiment of the above embodiments, a user pose-aware adaptive rendering system for light field display is provided. This system is implemented based on any of the user pose-aware adaptive rendering methods for light field display described above, and includes: The image acquisition and pose conversion module is used to acquire user head pose data through multi-source data acquisition and calculate the three-dimensional position coordinates of the user head pose data in the coordinate system of the light field display terminal. The priority calculation module is used to calculate the user's target viewing angle relative to the light field display terminal based on the three-dimensional position coordinates, and to calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle. The viewpoint segmentation module is used to divide each discrete display viewpoint into a focus area, a transition area, and an outer area based on the obtained priority value and a pre-set priority threshold, and to construct a high-priority viewpoint set with the focus area and a low-priority viewpoint set with the transition area and the outer area. The image rendering generation module is used to perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene, with a pre-set rendering target resolution as the benchmark, to obtain a first rendering image set; to perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene, to obtain a second rendering image set; and to generate a third rendering image set based on the first and second rendering image sets and the discrete display view in the outer area of ​​the low-priority view set through image deformation. The image output module is used to combine the obtained first, second, and third rendered image sets according to a preset viewpoint arrangement format and output the final rendered image.

[0055] Specifically, the implementation principle of the above embodiments is as follows: First, the image acquisition and pose conversion module deployed in the front field of the terminal acquires the dynamic feature data of the user in the physical space in real time through multi-source visual sensors (including facial contour and key feature point data). In order to ensure the robustness of the underlying data under complex ambient light and transient motion, the module maps the multi-source visual data into a six-degree-of-freedom pose in the sensor coordinate system through the perspective n-point (PnP) algorithm. Furthermore, it uses a pre-calibrated rigid body transformation matrix and Kalman filter temporal smoothing technology to calculate the smooth, continuous and globally spatial referenced three-dimensional absolute position coordinates of the user in the light field display panel coordinate system. Then, after receiving the above three-dimensional position coordinates, the priority calculation and viewpoint segmentation module calculates the target observation angle reflecting the user's true line of sight based on the built-in spatial geometric projection algorithm, and calculates the spatial angular distance between the dozens of discrete viewpoints fixed in the light field panel hardware and the observation angle. It then uses a mathematical model that integrates Gaussian decay functions to construct a priority physical field that reflects the global visual sensitivity of the viewpoint. Next, combined with the dynamic threshold boundary allocated by the current system operating state, all discrete viewpoints are precisely segmented and clustered into focal areas, transition areas, and peripheral areas, thereby constructing a structured set of high and low priority viewpoints. Next, the image rendering generation module extracts the partition set instructions output by the above modules, and uses its internally integrated cross-modal hybrid rendering engine to dynamically generate differentiated calculation instructions for each partition and send them to the GPU rendering pipeline. Specifically, the engine performs full-quality independent rendering by waking up the complete 3D lighting pipeline in the focal area, and performs degraded rendering by issuing lightweight instructions to downsample and cull complex lighting in the transition area. At the same time, based on the rendering process of the former two, the module intercepts and extracts the output color map and depth buffer (Z-buffer) in real time, and uses the built-in depth-based image deformation (DIBR) inference operator to directly perform 2D pixel-level spatial remapping on the massive redundant viewpoints classified as the peripheral area, in order to replace the traditional high-energy-consuming 3D rasterization calculation, thereby generating a complete cross-view rendering image set. Finally, the image output module receives the first, second and third rendered image sets output by the above-mentioned hybrid rendering engine, analyzes the reuse requirements of the physical optical lens array of the panel through the built-in light field underlying driving engine, and performs spatial interweaving and encoding recombination of the sub-pixels of each view according to the preset view arrangement format (such as Quilting layout), and projects naked-eye 3D panoramic images with real physical depth to the user intuitively through the high-speed display interface. Furthermore, during the entire system's loop rendering process, a dynamic computing power monitoring and timing gating logic module is deployed at the system's underlying layer. This module utilizes a built-in GPU performance counter to evaluate the millisecond-level frame rendering computing power reserve in real time. Once a computing power deficit is detected, an adaptive frequency reduction command is immediately triggered, shrinking the viewport from the edge to the core. Moreover, this module uses a supported historical sequence extrapolation algorithm to predict the user's future motion trajectory and, in conjunction with the hysteresis anti-shake threshold at the viewport boundary, pre-renders the predicted target viewpoint image to the dual buffer during the current frame's rendering idle time. This ensures, at the physical computing architecture level, the light field display system achieves perfect coordination between extreme smoothness and flicker-free operation under the dual constraints of severe user shaking and underlying computing power fluctuations. Example 3 Based on Examples 1 and 2, there is a practical application scenario for user pose image rendering on a desktop light field display terminal. This scenario employs a user pose-aware adaptive rendering method and system for light field display based on the user pose perception described in the above embodiments. Specifically, this embodiment uses a desktop light field display terminal equipped with an RK3588 on-chip system as the implementation carrier. The light field display panel of this terminal provides 16 discrete display viewing angles, with a horizontal viewing angle range of [missing information]. to angular spacing between adjacent viewing angles The single-view resolution is Pixels. The image acquisition device is an RGB camera fixedly mounted in the center of the top bezel of the terminal, with the lens facing the user, and the acquisition resolution is... Pixels, frame rate 30 frames per second.

[0056] Specifically, an adaptive rendering method for light field display based on user pose awareness includes the following steps: Step 1: Obtaining User Space Pose like Figure 1 As shown, the camera continuously captures user facial images. In this embodiment, a lightweight face detection model is used to extract facial key points from the image as user feature information. Six key points are selected: the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, the right corner of the mouth, and the chin point for pose solving. The preset reference model adopts a standard 3D face reference model, defining the relative coordinates of the above 6 key points in 3D space, using the 3D coordinates of each key point in the 3D face reference model. Two-dimensional coordinates of corresponding key points detected in the image As input, the pose of the user's head relative to the camera is solved using the perspective n-point (PnP) algorithm, based on the following projection relationship: ;in, The camera intrinsic parameter matrix is ​​pre-acquired and stored using a checkerboard calibration method. for Rotation matrix; for Translation vector; is the scale factor.

[0057] The rotation matrix obtained from the solution Translation vector Calculate the three-dimensional coordinates of the midpoint of the user's eyes in the camera coordinate system. The spatial transformation matrix is ​​then obtained through pre-calibration. After transforming to the coordinate system of the light field display panel, the three-dimensional coordinates of the midpoints of the user's eyes in the light field display panel coordinate system are obtained, expressed as follows: The panel coordinate system has its origin at the center of the panel display area. The axis moves to the right along the horizontal direction of the panel. The axis extends vertically upwards along the panel. The axis points towards the user along the normal direction of the panel, and after transformation, the coordinates of the user's observation point are obtained. .

[0058] Specifically, spatial transformation matrix The calibration method is as follows: Place a checkerboard calibration board of known dimensions flat on the surface of the light field display panel, aligning the calibration board coordinate system with the panel coordinate system. The camera captures an image of the calibration board from its installation position. Using the Zhang Zhengyou calibration method, both the camera's intrinsic parameters and extrinsic parameters relative to the calibration board are simultaneously obtained, thus determining the rigid body transformation matrix from the camera coordinate system to the panel coordinate system. This calibration only needs to be performed once before the terminal leaves the factory or during its first use.

[0059] To suppress single-frame detection jitter, an extended Kalman filter is applied to the transformed 3D position. The filter state vector is defined as: ;in, Indicates the first The filter state vector at time t. Let represent the three position components of the user's observation point in the panel coordinate system, and let represent the three velocity components of the user's observation point in the panel coordinate system. Under the assumption of a constant velocity motion model, the state transition matrix is ​​expressed as: ;in, Represents a 3rd order identity matrix. This represents the inter-frame time interval (33.3 milliseconds in this embodiment). The standard deviation of the position component in the process noise covariance matrix is ​​2 mm, and the standard deviation of the velocity component is 20 mm / s; the standard deviation of each position component in the observation noise covariance matrix is ​​5 mm, consistent with the typical estimation accuracy of the PnP algorithm at working distances of 500 mm to 800 mm. The filtered pose update rate remains at 30 Hz, with an end-to-end delay of approximately 25 milliseconds.

[0060] Specifically, in other implementations, user feature information may also be facial contours, eye center positions, or head and shoulder region features, and the corresponding preset reference model may be a statistical face shape model or a facial contour geometric model. Temporal smoothing may also employ standard Kalman filtering or moving average filtering.

[0061] Step 2: Calculation of target observation angle and viewpoint priority: Based on the user's position in the panel coordinate system described above, and based on the horizontal coordinates of the user's observation point in the panel coordinate system... and normal direction coordinates Calculate the target observation angle; the formula is as follows: For horizontal parallax-type light field display panels, the horizontal direction is the parallax distribution direction and the main dimension for distinguishing different discrete viewing angles. (Panel 1) The corresponding angles for each discrete display viewing angle are: In this embodiment, , , Inter-angular spacing .

[0062] Priority values ​​for each viewpoint are calculated using a Gaussian function: ; For the attenuation parameter, this embodiment takes... (Approximately 1.5 times the angular spacing) causes the viewing angle priority value to rapidly decay to below 0.15 for viewing angles more than two angular spacings away from the target.

[0063] Specifically, in other implementations, the priority function may also be an exponential decay function. ,in The attenuation constant; or a piecewise linear function. ,in To truncate the angle, when the angular distance exceeds... The priority value drops to zero. A common requirement for different functions is that they monotonically decrease with angular distance and reach their maximum value at the target observation angle.

[0064] Step 3: Viewpoint Division and Hierarchical Image Generation: Set a high threshold and low threshold The 16 discrete display viewpoints are divided into three regions. Priority value The perspective is brought into focus. Priority value The inclusion in the transition zone is lower than The focal area is categorized into the outer perimeter. The focal area constitutes the high-priority view set, while the transition area and the outer perimeter together constitute the low-priority view set.

[0065] Using the user's observation point coordinates at a certain moment For example, from equation (5) we get The optimal view index is determined by equation (6). ,correspond .set up , The results of the segmentation from each perspective are as follows: Focus area: , , There are 3 perspectives in total (with priority values ​​of approximately 0.90, 1.00, and 0.90 respectively). Transition zone: , , , There are a total of 4 perspectives; Outer area: to and There are a total of 9 perspectives.

[0066] Different image generation strategies are used for the three regions, specifically: Independent full-quality rendering is performed on the three perspectives in the focus area: resolution A complete PBR lighting pipeline (including ambient occlusion, real-time shadows, and specular reflections) with the highest level of detail for the geometric model. On an RK3588 Mali-G610 MC4 GPU, the latency is approximately 3.5 milliseconds per view and approximately 10.5 milliseconds for all three views combined. The four perspectives in the transition zone perform degraded independent rendering: (50% resolution) Rendered and then upsampled using bilinear interpolation to... Real-time shadows and ambient occlusion are disabled, while basic diffuse lighting is preserved at a medium level of geometric detail. The single-view time is approximately 1.2 milliseconds, and the total time for all four views is approximately 4.8 milliseconds. The nine viewpoints in the outer perimeter do not perform independent 3D scene rendering. The rendering is based on the already rendered viewpoint in the focal area. Using its color map and depth map as a reference viewpoint, images are generated for each peripheral viewpoint through depth-based image warping (DIBR). For each pixel in the reference viewpoint's color map... According to its depth value Baseline distance between the target viewpoint and the reference viewpoint Calculate the horizontal displacement: ;in The equivalent focal length parameter of the panel can be calculated from the panel viewing angle spacing and subpixel arrangement spacing: , This represents the sub-pixel spacing between adjacent viewpoints. It is calculated from the difference in angles between the two viewpoints and the nominal viewing distance.

[0067] Pixel remapping is implemented in the GPU fragment shader. After remapping, areas exposed due to changes in foreground occlusion will produce holes. In this embodiment, the holes are filled by horizontal background pixel extrapolation. When the angular distance between the target viewpoint and the reference viewpoint is large, deformation can also be performed simultaneously using a second reference viewpoint located on the other side of the target viewpoint. The results from both sides are then weighted and fused according to the inverse of the angular distance to reduce the hole area.

[0068] The single-view DIBR takes about 0.3 milliseconds, and the total time for 9 views is about 2.7 milliseconds.

[0069] The total time required to generate images from all 16 viewpoints is: With a frame cycle of less than 33.3 milliseconds, a stable output of 30 frames per second can be achieved. Compared to traditional equal-quality rendering solutions ( (milliseconds, approximately 18 frames per second), reducing total time by approximately 68%.

[0070] Step 4: Dynamic Adjustment of Computing Power Budget like Figure 2 As shown, at the beginning of each frame, the system estimates the effective processing time that the GPU can allocate to the current frame as a computing power budget by using GPU performance counters or frame time statistics (taking the moving average of the rendering time of the previous 4 frames and comparing it with the difference between the current frame's V-Sync signal timestamp). In another implementation, available computing power can also be estimated by using the GPU load query interface provided by the operating system or by using CPU-side timers to count the time consumed at each stage of the rendering pipeline.

[0071] The total cost of generating the current frame's viewpoint image must satisfy: ;in , , These represent the number of viewpoints in the focus area, transition area, and peripheral area, respectively. Cost of full-quality rendering of a single viewpoint in the focus area; For the transition zone at quality level Single-view cost (under) Multiple discrete levels can be taken, for example Corresponding to 50% resolution, (corresponding to 25% resolution); Cost of single-view DIBR in the peripheral area; This represents the total computing power budget for the current frame.

[0072] When the system is concurrently performing other computing tasks, it causes... When reducing (e.g., from 33.3 milliseconds to 23.3 milliseconds), adjust in the following priority order: Level 1: Reduce the rendering quality level of the transition area, decreasing the resolution from 50% to 25%. The single-view time was reduced from 1.2 milliseconds to about 0.5 milliseconds, and the four-view time was reduced by about 2.8 milliseconds.

[0073] Level 2: If the constraints are still not met, increase the level. The threshold was increased from 0.8 to 0.85, the number of focal area viewpoints was reduced from 3 to 2, and the original edge viewpoints were downgraded to transition area processing.

[0074] The total time after adjustment is approximately Milliseconds, satisfying the constraints.

[0075] The optimal viewing angle that is closest to the target's observation angle Always maintain full-quality rendering; this constraint is unaffected by adjustments to computing power.

[0076] when During recovery, recovery is performed in reverse order of quality level, with an interval of at least 2 frames between each recovery to avoid screen flickering caused by frequent switching of quality levels. For views near the partition boundary, hysteresis control is implemented: the priority value must be stable within the new interval for more than 3 consecutive frames before partition switching is performed to prevent frequent view jumps caused by slight changes in pose.

[0077] Step 5: Predictive Pre-rendering and Rollback: Using the position and velocity components in the state vector of the Kalman filter, the future... (In this embodiment, 200 milliseconds) The user's observation point position is linearly extrapolated for prediction, and the predicted target observation angle and the corresponding predicted optimal viewpoint index are calculated. If the predicted optimal viewpoint index is different from the current one, the focus area viewpoint corresponding to the predicted target viewpoint is pre-rendered during the idle period (approximately 15.3 milliseconds) after the current frame rendering is completed, and the result is written to the double buffer. If the actual pose and the predicted pose angle in the next frame deviate by more than [a certain value], [further action is taken]. The pre-rendered result is not used. At this point, a temporary image is generated using DIBR based on the existing reference viewpoint, and full-quality rendering of the correct viewpoint is started. The image is then replaced after the next frame is completed. The user-perceived switching delay is no more than one frame.

[0078] Step Six: Tracking and Handling Anomalies If the image acquisition device fails to detect a valid facial keypoint for 15 consecutive frames (0.5 seconds), the viewpoint segmentation scheme of the most recently detected valid one continues to generate output. If the detection failure lasts for more than 3 seconds, it switches to uniformly degraded rendering for all views (rendering all views independently at a reduced resolution) until the user pose is successfully detected again, at which point the adaptive partitioning strategy is restored. When multiple faces are detected simultaneously in the image, the face with the largest area is selected as the primary user. In another implementation, the poses of multiple users can be solved separately, and their spatial center can be taken as the virtual observation point.

[0079] Step 7: Output: After generating all viewpoint images, combine and output them according to the viewpoint arrangement format required by the panel. This example is... Quilting layout. Other implementations may also use an interlaced layout. Performance comparison data between traditional equal-quality rendering schemes and the scheme of this invention are shown in Table 1.

[0080] Table 1 Performance Comparison Data

[0081] The above data represents test results on the RK3588 platform. Actual performance may fluctuate due to differences in scene complexity and driver implementation. The load reduction is greater in the 32-view configuration because the peripheral view area accounts for a higher proportion, allowing for more independent rendering that can be replaced by DIBR.

[0082] Example 4

[0083] like Figure 3 As shown, this embodiment proposes a terminal device for a user pose-aware light field display adaptive rendering method. The terminal device includes an external device, at least one memory, at least one processor, and a bus connecting different platform systems.

[0084] The memory may include readable media in the form of volatile memory, such as RAM 211 and / or cache memory, and may further include ROM 213.

[0085] The memory also stores a computer program that can be executed by a processor, causing the processor to perform any of the user pose-aware adaptive rendering methods for light field display described in the embodiments of this application. The specific implementation and technical effects are consistent with those described in the embodiments of the above methods, and some details will not be repeated here. The memory may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0086] Accordingly, the processor can execute the aforementioned computer programs, as well as executable programs / utilities.

[0087] A bus can represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus structures.

[0088] External devices may include a light field display panel that can communicate with the terminal device for outputting rendered images; the light field display panel may integrate an image acquisition device for capturing user images. Furthermore, the terminal device can communicate with one or more other external devices, such as a keyboard, pointing device, Bluetooth device, etc., and with one or more devices capable of interacting with the terminal device, and / or with any device that enables the terminal device to communicate with one or more other computing devices (e.g., a router, modem, etc.). This communication can be performed through I / O interfaces. Additionally, the terminal device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter can communicate with other modules of the terminal device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the terminal device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0089] Example 5

[0090] like Figure 4As shown, based on Embodiment 1, this embodiment proposes a computer-readable storage medium for a user pose-aware adaptive rendering method for light field display. The computer-readable storage medium stores instructions that, when executed by a processor, implement any of the aforementioned user pose-aware adaptive rendering methods for light field display. Its specific implementation method is consistent with the implementation methods and achieved technical effects described in the embodiments above, and some details will not be repeated.

[0091] This embodiment provides a program product for implementing the above-described method, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this embodiment, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0092] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to user computing devices via any type of network, including local area networks (LANs) or wide area networks (WANs), or they can be connected to external computing devices (e.g., via the Internet through an Internet service provider).

[0093] This invention is described from the perspectives of its intended use, effectiveness, progress, and novelty. Its practical and progressive features meet the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings are merely preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc., that are similar to or identical to those of this application, i.e., all equivalent substitutions or modifications made in accordance with the scope of this patent application, shall fall within the scope of protection of this patent application.

[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A user pose-aware adaptive rendering method for light field display, characterized in that, Includes the following steps: S1. Acquire user head pose data through multi-source data acquisition, and calculate the three-dimensional position coordinates of user head pose data in the coordinate system of the light field display terminal; S2. Calculate the target viewing angle of the user relative to the light field display terminal based on the three-dimensional position coordinates, and calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle; S3. Based on the obtained priority values ​​and combined with the pre-set priority thresholds, each discrete display viewpoint is divided into a focus area, a transition area, and an outer area. A high-priority viewpoint set is constructed using the focus area, and a low-priority viewpoint set is constructed using the transition area and the outer area. S4. Based on the pre-set target resolution, perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene to obtain the first rendering image set. Perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene to obtain the second rendering image set. Based on the first rendering image set and the second rendering image set, generate the third rendering image set by image deformation for each discrete display view in the outer area of ​​the low-priority view set. S5. Based on the obtained first, second, and third rendered image sets, combine them according to a preset viewpoint format and output the final rendered image.

2. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, Step S1 includes the following sub-steps: S101. Acquire user images through an image acquisition device deployed within the light field display terminal, and extract user feature information, wherein the user feature information includes facial key point information; S102. Based on user feature information and combined with a preset reference model, solve the pose parameters of the user's head relative to the image acquisition device to obtain user head pose data; S103. Based on the user's head pose data, the user's head pose data is mapped into three-dimensional position coordinates of the user's head in the coordinate system of the light field display terminal through the spatial transformation relationship between the pre-calibrated coordinate system of the image acquisition device and the coordinate system of the light field display panel, and then time-series smoothing is performed through Kalman filtering or extended Kalman filtering.

3. The adaptive rendering method for light field display based on user pose awareness according to claim 2, characterized in that, Step S102 includes the following sub-steps: S1021. Obtain a three-dimensional face reference model as a reference model, and based on the correspondence between the three-dimensional coordinates of each key point in the three-dimensional face reference model and the two-dimensional coordinates of the corresponding key points in the user feature information, use the perspective n-point algorithm to solve the rotation parameters and translation parameters. S1022. Determine the three-dimensional coordinates of the midpoint of the user's eyes in the coordinate system of the image acquisition device based on the rotation and translation parameters, and use them as the position coordinates of the user's head in the coordinate system of the image acquisition device.

4. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, Step S2 includes the following steps: S201: Calculate the target viewing angle of the user's head relative to the light field display terminal based on the three-dimensional position coordinates obtained in step S1; S202: Based on the target viewing angle, calculate the angular distance between the angle corresponding to each discrete display viewing angle of the light field display terminal and the angle corresponding to the target viewing angle. Substitute the angular distance as an independent variable into the priority function to generate the priority value for each discrete display viewing angle. The priority function is expressed as follows: ； in, This represents the priority value for each discrete display viewpoint. Indicates the first Each discrete display viewing angle corresponds to an angle. Indicates the target viewing angle. Indicates the attenuation parameter. This represents an exponential function.

5. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, Step S3 includes the following sub-steps: S301. Based on the actual working parameters of the light field display terminal, set high-priority threshold and low-priority threshold; S302. Mark the viewpoints with priority values ​​not lower than the high priority threshold as the focus area, and construct a high priority viewpoint set with all discrete display viewpoints within the focus area; S303. Views with priority values ​​lower than the high priority threshold but not lower than the low priority threshold are marked as transition zones, and views with priority values ​​lower than the low threshold are marked as peripheral zones. A set of low priority views is constructed using all discrete display views within the transition zone and peripheral zone.

6. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, Step S4 includes the following sub-steps: S401. Obtain the high-priority view set, the low-priority view image set, and the preset rendering target resolution; S402. Based on the high-priority view set, and taking the rendering target resolution as a benchmark, combine the complete lighting calculation pipeline and perform full-quality independent 3D rendering on each discrete display view within the high-priority view set to obtain the first rendering image set. S403. Based on the low-priority view set, perform degraded independent rendering on each discrete display view marked as the transition zone within the low-priority view set to obtain a second rendering image set. The degraded independent rendering includes at least rendering at a resolution lower than the rendering target resolution and then upsampling to the rendering target resolution, reducing the geometric detail level, turning off real-time shadow calculation, turning off ambient occlusion, or reducing the number of illumination samplings. S404. Based on the low-priority viewpoint set, each discrete display viewpoint marked as the peripheral area within the low-priority viewpoint set is treated as a part that does not perform independent 3D rendering. Using the color map and depth map of the already generated rendered images in the first and second rendered image sets as references, a third rendered image set is generated through depth-based image deformation. The depth-based image deformation is represented as follows: ； in, Represents pixels The horizontal displacement relative to the reference viewpoint image in the target viewpoint image. This represents the equivalent focal length parameter of the light field display terminal. This represents the baseline distance between the target viewpoint and the reference viewpoint. Represents pixels in the depth map of the reference view. The depth value at that location.

7. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, The following steps are included before performing step S4: A1. Real-time acquisition of available computing power for image rendering in the current frame, dynamic adjustment of the division boundary between the high-priority view set and the low-priority view set and / or the rendering quality level of each view, so that the total cost of image generation for all views in the current frame does not exceed the preset computing power budget. A2. When the real-time available computing power of the system decreases, reduce the rendering quality level of the viewpoints in the low-priority viewpoint set. If the constraints are still not met, narrow the range of the high-priority viewpoint set, and mark the discrete display viewpoint that is closest to the target observation angle as the optimal viewpoint, and always keep it rendered independently.

8. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, It also includes extrapolating and predicting the user's observation point position within a future preset time window based on the user's historical pose sequence, and calculating the predicted target observation angle; when the optimal viewpoint corresponding to the predicted target observation angle is different from the current optimal viewpoint, pre-rendering is performed on at least some high-priority viewpoints corresponding to the predicted target observation angle during the rendering idle period of the current frame and the results are written to the cache; when the angle deviation between the actual user pose and the predicted pose in the next frame exceeds a preset deviation threshold, a temporary image generated by image deformation based on the current existing reference viewpoint is output first, and the temporary image is replaced by an independent rendering result in subsequent frames.

9. The adaptive rendering method for light field display based on user pose awareness according to claim 1, characterized in that, When the image acquisition device fails to detect valid user feature information within a preset number of consecutive frames, it continues to generate images using the viewpoint segmentation scheme from the most recent valid detection. When the duration of the failure to detect valid user feature information exceeds a preset timeout threshold, it switches to uniformly degraded rendering for all discrete display viewpoints. Hysteresis control is performed on the segmentation boundary between consecutive frames. When a viewpoint with a priority value near the segmentation threshold undergoes a change in its set affiliation, its priority value must remain within the threshold range corresponding to the new set for at least a preset number of consecutive frames before set switching can be performed.

10. A user pose-aware adaptive rendering system for light field display, the system being implemented based on the user pose-aware adaptive rendering method for light field display according to any one of claims 1-9, characterized in that, include: The image acquisition and pose conversion module is used to acquire user head pose data through multi-source data acquisition and calculate the three-dimensional position coordinates of the user head pose data in the coordinate system of the light field display terminal. The priority calculation module is used to calculate the user's target viewing angle relative to the light field display terminal based on the three-dimensional position coordinates, and to calculate the priority value of each discrete display viewing angle based on the angular distance between each discrete display viewing angle of the light field display terminal and the target viewing angle. The viewpoint segmentation module is used to divide each discrete display viewpoint into a focus area, a transition area, and an outer area based on the obtained priority value and a pre-set priority threshold, and to construct a high-priority viewpoint set with the focus area and a low-priority viewpoint set with the transition area and the outer area. The image rendering generation module is used to perform full-quality independent rendering of each discrete display view in the high-priority view set based on the three-dimensional scene, with a pre-set rendering target resolution as the benchmark, to obtain a first rendering image set; to perform degraded independent rendering of each discrete display view in the transition area of ​​the low-priority view set based on the three-dimensional scene, to obtain a second rendering image set; and to generate a third rendering image set based on the first and second rendering image sets and the discrete display view in the outer area of ​​the low-priority view set through image deformation. The image output module is used to combine the obtained first, second, and third rendered image sets according to a preset viewpoint arrangement format and output the final rendered image.

Images