An immersive display system and method for ancient book culture
By establishing multiple quality levels of 3D models on mobile terminals and dynamically adjusting them based on user interaction status and rendering load, the Lagrange multipliers are adaptively updated, solving the problem of uneven allocation of 3D model rendering resources on mobile terminals and achieving efficient rendering resource scheduling and smoothness.
Patent Information
- Application Number
- CN202611619056.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-20
- Publication Date
- 2026-11-17
Smart Images

Figure CN123118874A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D graphics rendering and adaptive scheduling technology. More specifically, this invention relates to a method and system for immersive display of ancient books and culture. Background Technology
[0002] With the deepening application of digital technology in the protection and dissemination of cultural heritage, digitally reconstructing cultural images recorded in ancient books through 3D modeling technology and displaying them immersively on mobile terminals has become an important way to revitalize and utilize culture. When users perform interactive operations such as rotating, scaling, and shape switching on 3D models through mobile terminals, the system must constantly seek a balance between image detail and smoothness due to the limited computing and rendering hardware resources of mobile terminals. Basic existing technologies usually use fixed-precision mesh models for rendering. This fixed rendering precision strategy ignores the performance differences and operating status of terminal devices. Due to the lack of a dynamic scheduling mechanism, the fixed rendering strategy easily leads to screen stuttering and frame rate drops on low-end devices when loading complex models due to insufficient computing power. At the same time, it prevents high-end devices with a large amount of graphics processing margin from providing richer image details, resulting in a serious imbalance in the allocation of rendering resources and limited device applicability.
[0003] To address the resource allocation imbalance caused by the aforementioned fixed rendering strategy, Chinese patent application CN114332405A discloses a real-time display method for ultra-high-definition 3D scanned models of cultural relics. This method obtains an ultra-high-definition triangular mesh after 3D scanning of the cultural relic, breaks it down into hierarchical clusters composed of triangles, analyzes the rendering accuracy coefficient based on the model's screen proportion, calculates the surface accuracy of the triangular proxy objects of the model, quantizes the mesh vertex positions of the hierarchical clusters, and the display engine judges the display status in real time during the display process. It also dynamically switches the detail level of the proxy model clusters during runtime to achieve real-time display.
[0004] However, when switching between detail levels, the application mainly relies on the static determination of the model's screen area and rendering accuracy coefficient, without incorporating the user's perception characteristics in real-time interaction and the dynamic changes in the terminal's frame-by-frame rendering load into the joint decision-making of level selection. When the user quickly rotates or switches between models of different complexities, the fixed accuracy determination strategy cannot adaptively adjust the level switching intensity according to the inter-frame changes in rendering time, resulting in frame rate fluctuations on low-performance terminals and the rendering capabilities of high-performance terminals not being fully utilized. Summary of the Invention
[0005] To address the technical problem in existing technologies where mobile terminals lack detailed-level adaptive scheduling for 3D model rendering based on joint decision-making of real-time interactive status and frame-by-frame rendering load, this invention provides a method and system for immersive display of ancient book culture.
[0006] In a first aspect, the present invention provides an immersive display method for ancient book culture, comprising: S1, establishing multiple quality levels for each order model, and obtaining the rendering time cost and visual distortion of each quality level; S2, collecting user interaction status before rendering each frame, calculating the perceptual masking factor, and calculating the ratio of the actual rendering time of the current frame to that of the previous frame to obtain the frame time jump ratio; S3, adaptively updating the Lagrange multiplier through a feedforward channel and a feedback channel, wherein when an order switching event is detected, the feedforward channel presets the Lagrange multiplier as the convergence reference value of the target order, and the convergence parameter... The evaluation value is the weighted average of the historical convergence values of the Lagrange multiplier at this order state; the feedback channel calculates the step size amplification factor based on the frame time abrupt change ratio, and adjusts the update step size of the Lagrange multiplier with the step size amplification factor. The step size amplification factor is positively correlated with the degree of deviation of the frame time abrupt change ratio from 1; S4, based on the updated Lagrange multiplier, perceptual masking factor, rendering time cost and visual distortion, the optimal quality level of the current frame is solved through the rate distortion cost function; S5, after applying inter-frame smoothing constraints to the optimal quality level, rendering is performed, and the actual rendering time is fed back to the next frame.
[0007] This invention establishes multiple quality levels for each state model and quantifies rendering time cost and visual distortion. Before rendering each frame, it collects user interaction status to calculate the perceptual masking factor and the frame time abrupt change ratio. It then adaptively updates the Lagrange multiplier through feedforward and feedback channels. Finally, it combines the perceptual masking factor, rendering time cost, and visual distortion pass-through distortion cost function to solve for the optimal quality level of the current frame. Finally, it applies inter-frame smoothing constraints and executes rendering. This allows mobile terminals with different performance levels to adaptively select a quality level that balances visual quality and rendering smoothness in each state model and interaction state, reducing the stuttering on low-end devices and the resource waste on high-end devices caused by fixed rendering precision strategies.
[0008] Preferably, the step of collecting user interaction status and calculating the perceptual masking factor before rendering each frame includes: collecting user interaction status including rotation angular velocity and scaling rate; dividing the absolute value of rotation angular velocity by the angular velocity reference value to obtain a rotation normalized value, dividing the absolute value of scaling rate by the scaling rate reference value to obtain a scaling normalized value, and taking the larger value between the rotation normalized value and the scaling normalized value and adding 1 to obtain the perceptual masking factor.
[0009] Preferably, the method for updating the convergence reference value is as follows: each time the Lagrange multiplier confirms convergence at a certain order state, the Lagrange multiplier value at the current convergence time is taken as the current convergence value; the new convergence reference value is equal to the old convergence reference value multiplied by the retention coefficient plus the current convergence value multiplied by the update coefficient; wherein, the sum of the retention coefficient and the update coefficient is 1, and when the absolute value of the difference between the frame time change ratio of each frame within a consecutive preset number of frames and 1 is less than the preset fluctuation threshold, the Lagrange multiplier is determined to be in a convergence state.
[0010] Preferably, the feedforward channel includes deviation correction: after each step-state switch, the Lagrange multiplier is preset as the convergence reference value, and the difference between the Lagrange multiplier value at the time of the first confirmed convergence and the convergence reference value is divided by the convergence reference value to obtain the deviation correction coefficient; when the deviation correction coefficients obtained from consecutive step-state switches for a preset number of times all have the same sign, the product of the average of the deviation correction coefficients for the preset number of times and the current convergence reference value is added to the convergence reference value; wherein, the cumulative amount of the addition is less than or equal to a preset proportion of the convergence reference value, and the corrected convergence reference value is 0 when it is less than 0.
[0011] This invention obtains the deviation correction coefficient by dividing the difference between the Lagrange multiplier value at the time of initial convergence confirmation and the convergence reference value by the convergence reference value. When the values have consecutive same signs, bounded compensation is applied to the convergence reference value. This can track the gradual performance drift of the mobile terminal caused by factors such as temperature rise or background load changes, so that the feedforward preset accuracy continues to improve with running time. At the same time, the upper limit constraint of the cumulative amount prevents extreme compensation from causing the convergence reference value to deviate from the reasonable range.
[0012] Preferably, adjusting the update step size of the Lagrange multiplier with the step size amplification factor includes: multiplying the base step size, the step size amplification factor, and the frame time deviation to obtain the increment of the Lagrange multiplier; adding the increment of the Lagrange multiplier to the Lagrange multiplier to obtain the updated Lagrange multiplier; wherein, the lower limit of the updated Lagrange multiplier is 0, and the frame time deviation is the actual rendering time of the current frame minus the target frame time and then divided by the target frame time.
[0013] Preferably, the method for obtaining the step size amplification factor is as follows: take the absolute value of the frame time change ratio minus 1, divide it by the normal fluctuation range, and then add 1; wherein, a preset upper limit is set, and when the value of the calculated step size amplification factor exceeds the preset upper limit, the preset upper limit is taken as the step size amplification factor; otherwise, the value of the calculated step size amplification factor is taken as the step size amplification factor.
[0014] The step size amplification factor of this invention is calculated by dividing the absolute value of the frame time change ratio deviating from 1 by the normal fluctuation range and then adding 1, and a preset upper limit is set for truncation, so that the adjustment range of the feedback channel is proportionally amplified to the actual change degree. When the change is large, it responds quickly without overshooting and oscillating, and maintains small step size fine adjustment without introducing additional jitter when the operation is stable, thus taking into account both response speed and adjustment stability.
[0015] Preferably, the method for obtaining the normal fluctuation range is as follows: in a frame sequence in which the frame time change ratio of multiple consecutive frames is within a preset stable range, the standard deviation of the frame time change ratio is calculated, and the standard deviation of the frame time change ratio is used as the normal fluctuation range; wherein, when the system is initialized, a preset default value is taken as the normal fluctuation range, and after running a preset number of frames, the preset default value is replaced with the standard deviation of the measured frame time change ratio.
[0016] Preferably, the rate-distortion cost function calculates the cost value for each quality level, where the cost value is the sum of the perceptual distortion term and the rendering cost term; the perceptual distortion term is the visual distortion degree of that quality level divided by the perceptual masking factor; the rendering cost term is the product of the updated Lagrange multiplier and the rendering time cost of that quality level; the quality level with the lowest cost value among all quality levels is selected as the optimal quality level; if multiple quality levels have the same cost value, the quality level with the lower number is selected as the optimal quality level.
[0017] This invention uses the rate-distortion cost function to take the sum of the perceptual distortion term and the rendering cost term as a unified evaluation index for each quality level. After the visual distortion degree is normalized by the perceptual masking factor, it is added to the rendering time cost weighted by the Lagrange multiplier with the same dimension. This makes the quality level selection mathematically optimal under given frame rate constraints and user operation conditions, avoiding the subjectivity and scene limitations caused by manually setting the switching threshold.
[0018] Preferably, the inter-frame smoothing constraint includes: setting the optimal quality level of the current frame as the target level, and the quality level actually executed in the previous frame as the current level; when the target level is greater than the current level and the absolute value of the difference between the two exceeds a preset maximum jump amplitude, the current level is increased by the preset maximum jump amplitude and then used as the execution level; when the target level is less than the current level and the absolute value of the difference between the two exceeds the preset maximum jump amplitude, the current level is decreased by the preset maximum jump amplitude and then used as the execution level; when the absolute value of the difference between the target level and the current level does not exceed the preset maximum jump amplitude, the target level is directly determined as the execution level; when a state switching event occurs, the current level is reset to the quality level obtained by solving the rate-distortion cost function in the new state.
[0019] This invention limits the change in quality level between adjacent frames to a preset maximum jump range through inter-frame smoothing constraints, and resets the current level to the quality level solved by the rate-distortion cost function in the new level when switching between levels. This avoids screen flicker caused by large jumps in quality level between consecutive frames. At the same time, it ensures that the smoothing constraints after the level switch are based on the cost data of the new level itself rather than the residual level of the old level, thus guaranteeing visual continuity when switching between levels.
[0020] Secondly, the present invention provides an immersive display system for ancient book culture, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned immersive display method for ancient book culture is implemented.
[0021] By adopting the above technical solution, a computer program is generated from the above-mentioned immersive display method of ancient book culture and stored in the memory so that it can be loaded and executed by the processor. In this way, a terminal device can be made based on the memory and the processor for convenient use.
[0022] The technical solution of the present invention has the following beneficial technical effects: This invention establishes multiple quality levels for each state model and obtains the rendering time cost and visual distortion for each quality level, providing rendering time cost and visual distortion for solving the optimal quality level. This invention directly pre-sets the Lagrange multiplier to near the historical convergence value of the target state through a feedforward channel for the predictable event of state switching. Simultaneously, it adaptively adjusts the step size of unpredictable rendering load changes through the step size amplification coefficient of the feedback channel, enabling the Lagrange multiplier to converge to a new equilibrium value within a few frames after various state changes. Compared to a fixed-step, frame-by-frame gradual adjustment method, the frame rate recovery speed is improved. Furthermore, the rate-distortion cost function incorporates the perceptual masking factor into the normalization calculation of the distortion term, allowing for a reduction in rendering precision without perceptible during rapid user operations, and automatically restoring high image quality when the user is stationary, achieving perceptually lossless dynamic quality scheduling. Attached Figure Description
[0023] Figure 1 This is a flowchart of an immersive display method for ancient book culture according to the present invention; Figure 2 This is the feedforward-feedback dual-channel Lagrange multiplier update logic diagram of the present invention; Figure 3 This is a quantitative model diagram of visual distortion and rendering time cost at various quality levels in this invention; Figure 4 This is a graph comparing the rendering performance optimization of the control group and the experimental group. Detailed Implementation
[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0025] This invention discloses an immersive display method for ancient book culture, referring to... Figure 1 This includes the following steps: S1. Obtain the rendering time cost and visual distortion for each quality level. Specifically, establish multiple quality levels for each stage model and obtain the rendering time cost and visual distortion for each quality level.
[0026] The input for each state model includes model mesh data, material data, and the corresponding state number. Multiple quality levels are established based on model complexity, and these quality levels are numbered sequentially from low to high visual quality. There are four quality levels: Quality Level 1, Quality Level 2, Quality Level 3, and Quality Level 4. Quality Level 1 corresponds to the lowest model complexity, and Quality Level 4 corresponds to the highest model complexity. The number of quality levels can be adjusted between three and six depending on the mobile terminal's storage capacity and graphics processing capabilities; using four levels maintains a balance between quality variation and scheduling granularity.
[0027] A quadratic error metric mesh simplification method is used to simplify the mesh data of each order-state model, resulting in mesh data of different quality levels with varying mesh sizes. Quality level 4 retains the original mesh data; quality level 3 has half the number of faces of the original mesh; quality level 2 has one-quarter the number of faces; and quality level 1 has one-eighth the number of faces. During the simplification process, the correspondence between the order-state number and the mesh data for each quality level is preserved, allowing subsequent scheduling to directly obtain the corresponding mesh data based on the quality level. The quality level is determined by the following criteria: the corresponding mesh data can be loaded and rendered independently, and the higher the quality level number, the more detail the model retains is not lower than that of the model with the lower quality level number. Once these conditions are met, multiple quality levels corresponding to each order-state model are output.
[0028] In an optional embodiment, the target operating device performs benchmark rendering tests on each quality level. The benchmark rendering tests measure the single-frame rendering time of the model data for each quality level to obtain the rendering time cost for each quality level. Before the test, the model data is loaded, and one frame is used as a warm-up frame. Then, five frames are rendered continuously under the same viewpoint, display resolution, and lighting conditions. The time difference from the start of rendering to completion for each frame is recorded. The average of the five valid measurement results is taken as the rendering time cost for the corresponding quality level, in milliseconds. If, during the test, resources are loaded for the first time, the application is switched to the background, or rendering is interrupted, the corresponding frame is not considered a valid measurement result, and measurements are supplemented until five valid results are obtained. The rendering time cost for each quality level is saved separately according to the stage number, quality level number, and target operating device, and the rendering time cost is output.
[0029] The input for visual distortion is model data for quality level 4, model data for the quality level to be tested, and a uniform viewing angle. Using screen-space pixel error, under fixed display and rendering parameters of the target device, the models for each quality level are compared with the highest quality level model to obtain the visual distortion corresponding to each quality level. During the comparison, the viewing distance is set to 2.5 times the diameter of the model's bounding sphere. Quality level 4 and the quality level to be tested are rendered using the same projection parameters and display resolution. Each visible vertex of the quality level to be tested is projected onto screen pixel coordinates, and the screen pixel coordinates of the corresponding position of that vertex on the surface of the quality level 4 model are obtained. The pixel deviation is obtained by taking the square root of the sum of the squares of the lateral and vertical deviations between the vertex of the quality level to be tested and its corresponding position on the surface of the quality level 4 model. The maximum pixel deviation among all valid corresponding points is taken as the visual distortion for the corresponding quality level. When comparing quality level 4 with itself, the visual distortion is set to 0 pixels; for other quality levels, the maximum measured pixel deviation is recorded. If the corresponding point of the quality level to be measured does not fall within the display area, the corresponding point will not be included in the statistical results of the current viewpoint. When the number of valid corresponding points is less than the preset statistical number, another preset observation viewpoint at the same viewing distance will be used to supplement the measurement until the complete visual distortion is obtained.
[0030] Due to differences in geometric complexity, detail distribution, and texture features among different model stages, a separate mapping relationship is established for quality level, rendering time cost, and visual distortion for each model stage, and measurement results are not shared between different models. For each model stage of ancient mythical beasts such as Bai Ze, a correspondence between quality level, rendering time cost, and visual distortion is established in the aforementioned manner. The correspondence includes at least the stage number, quality level number, rendering time cost, and visual distortion, and measurements are taken under the same target operating device, the same display resolution, the same viewing point, and the same rendering conditions. Subsequent quality level selections use the corresponding data from the same stage stage, and measurement results from other stages or other target operating devices are not directly substituted.
[0031] The number of quality levels, the ratio of mesh faces, the number of warm-up frames, the number of effective measurement frames, and the standard viewing distance are all configurable parameters.
[0032] See Figure 3 Taking a 3D model of a mythical beast from an ancient book as an example, four quality levels were used. The original model had 30,000 faces. After offline simplification, quality level 4 maintained 30,000 faces, quality level 3 had 15,000 faces, quality level 2 had 7,500 faces, and quality level 1 had 3,750 faces. This means the mesh face ratios were 1, 1 / 2, 1 / 4, and 1 / 8 respectively. One warm-up frame was used, and five effective measurement frames were used. The standard viewing distance was 2.5 times the diameter of the model's bounding sphere. The test results were: quality level 1 had a visual distortion of 2.8 pixels and a rendering time of 3 milliseconds; quality level 2 had 1.2 pixels and 5.1 milliseconds; quality level 3 had 0.4 pixels and 8.6 milliseconds; and quality level 4 had 0 pixels and 14.2 milliseconds.
[0033] The number of quality levels is determined based on the scheduling overhead that the mobile terminal can tolerate; the ratio of mesh faces is determined based on the visual differences between adjacent quality levels; the number of measurement frames is determined based on the need to remove single-frame timing jitter; and the standard viewing distance is 2.5 times the diameter of the model's bounding sphere. This hierarchical processing provides each level model with directly comparable rendering time costs and visual distortion, giving clear data basis for subsequent quality level selection. Furthermore, the gradual changes between quality levels preserve continuous adjustment space for inter-frame smoothing.
[0034] S2. Calculate the perceptual masking factor and frame time jump ratio. Specifically, before rendering each frame, the user interaction state is collected, the perceptual masking factor is calculated, and the perceptual masking factor is positively correlated with the user's operation speed; the frame time jump ratio is obtained by calculating the ratio of the actual rendering time of the current frame to that of the previous frame.
[0035] The rendering program reads the user interaction state before rendering each frame. The user interaction state includes the model rotation angle, model scaling, current frame timestamp, previous frame timestamp, actual rendering time of the current frame, actual rendering time of the previous frame, and state transition events. The model rotation angle is generated by user touch dragging or other rotation operations; the model scaling is generated by user scaling operations; and the state transition event is generated when a change in the target state number of the currently displayed object is detected. The target state number can be generated by user interaction operations or by the rendering program according to preset switching rules. The program checks the validity of the read data. When the timestamps are consecutive and the rotation angle and scaling are all valid values, the current user interaction state is output. When any data is missing or the timestamp order is abnormal, the most recent valid user interaction state is retained, and invalid time ratios are treated as if there were no sudden changes.
[0036] The rotational angular velocity is obtained by processing the model's rotation angle using the adjacent frame difference method. Specifically, the model's rotation angle in the current frame is subtracted from the model's rotation angle in the previous frame, and then divided by the time interval between the current and previous frames. The result is expressed in degrees per second. The scaling rate is obtained by processing the model's scaling using the adjacent frame difference method. The difference in view distance between the current and previous frames is divided by the time interval between the current and previous frames. Both the rotational angular velocity and scaling rate retain their positive and negative directions, which are only used to indicate the direction of motion. Their absolute values are used when calculating the perceptual masking factor later. The actual rendering time of the current frame is obtained by recording the difference in timestamps from the start to the end of the previous frame's rendering. Whether a state transition event has occurred and the target state number are obtained by listening to the state upgrade button click event on the user interface.
[0037] When the time interval between adjacent frames is not less than 2ms, the difference result is used as the rotation angular velocity and scaling rate. When the time interval between adjacent frames is less than 2ms, the time interval is 0, or the timestamp is invalid, the current difference result is not used; instead, the most recent valid rotation angular velocity and scaling rate are used. The preset minimum effective frame interval is 2ms in this embodiment, which can be configured within the range of 1ms to 5ms depending on the system timestamp accuracy. The value of the minimum effective frame interval is used to eliminate the influence of timestamp accuracy errors on speed calculation. Under common 60fps display conditions, the normal frame interval is approximately 16.67ms, and the 2ms interval determination condition will not truncate normal interactive data.
[0038] In an optional embodiment, the perceptual masking factor of the current frame is calculated based on the rotational angular velocity and the scaling rate. Specifically, the inputs to the perceptual masking factor are the rotational angular velocity, the scaling rate, a reference value for the angular velocity, and a reference value for the scaling rate. Based on the physiological characteristic of the human visual system that its sensitivity to static details decreases in dynamic scenes, the perceptual masking factor is calculated as follows: the absolute value of the rotational angular velocity is divided by the angular velocity reference value to obtain the rotational normalized value, and the absolute value of the scaling rate is divided by the scaling rate reference value to obtain the scaling normalized value. The larger of the rotational normalized value and the scaling normalized value is then added to obtain the perceptual masking factor. The minimum value of the perceptual masking factor is 1; a larger value indicates lower human visual perception and a higher tolerance for accuracy degradation.
[0039] When the user does not rotate or scale the model, both normalized values are 0, and the perceptual masking factor is 1. As the rotational angular velocity or scaling rate increases, at least one normalized value increases, and the perceptual masking factor increases accordingly.
[0040] The angular velocity reference value is a calibration parameter used to characterize the user's sensitivity to changes in model rotation details. Its specific value is determined based on the target mobile terminal's display size, refresh rate, viewing distance, and user subjective evaluation experiments. In this embodiment, the value ranges from 45° / s to 90° / s. Research on the human eye's motion blur perception threshold shows that when an object's speed on the screen exceeds 60 degrees per second, the human eye's ability to distinguish details decreases to half of its static resolution. Therefore, a preferred angular velocity reference value is 60° / s; rotating the model at 60° / s will result in a rotation normalization value of 1. The scaling rate reference value uses similar logic, taking the rate of change in viewing distance as equal to the model's bounding sphere diameter per second. That is, a preferred scaling rate reference value is 1 times the model's bounding sphere diameter per second, with a range of 0.5 to 2 times the model's bounding sphere diameter per second. The angular velocity reference value is determined based on the user's perceptible change in rotation details within the mobile terminal's display area. The scaling rate reference value is determined based on the user's common operating speed for changes in model viewing distance. In actual deployment, subjective evaluation data of image detail by different users at different operating speeds can be collected. When the operating speed reaches a critical value where the image detail discernibility significantly decreases, the corresponding speed will be used as a reference value for angular velocity or scaling rate.
[0041] The frame time variation ratio is calculated by taking the actual rendering time of the current frame and the actual rendering time of the previous frame as input. The rendering process of adjacent frames is measured using a timestamp difference method to obtain the actual rendering time. Specifically, the timestamp when rendering begins and the timestamp when display data is submitted are recorded. The actual rendering time of the corresponding frame is obtained by subtracting the previous timestamp from the latter. The frame time variation ratio is then calculated by dividing the actual rendering time of the current frame by the actual rendering time of the previous frame. A frame time variation ratio of 1 indicates that the rendering time of adjacent frames has not changed; a frame time variation ratio greater than 1 indicates that the rendering time of the current frame has increased; and a frame time variation ratio less than 1 indicates that the rendering time of the current frame has decreased.
[0042] The actual rendering time of the current frame is written to the rendering time record after the current frame is completed, and used as the actual rendering time of the current frame before the next frame begins rendering; the previous rendering time record is used as the actual rendering time of the previous frame. When the first frame after the program starts has not yet formed the actual rendering time of the previous frame, the frame time mutation ratio is set to 1. When the actual rendering time of the current frame is less than or equal to 0, the actual rendering time of the previous frame is less than or equal to 0, or any rendering time record is invalid, the frame time mutation ratio is also set to 1, and this data is judged as having no valid mutation data. Otherwise, the frame time mutation ratio is output according to the ratio of the actual rendering time of the current frame to the actual rendering time of the previous frame.
[0043] The demonstration program writes the perception masking factor, frame latency abrupt change ratio, rotation angular velocity, scaling rate, and state transition events into the current frame's runtime data. After the data is written, the runtime data is only output to the subsequent scheduling process if both the perception masking factor and the frame latency abrupt change ratio have valid values. If the interactive data required for calculating the perception masking factor is invalid, the perception masking factor from the previous frame is used; if no valid value from the previous frame exists, the perception masking factor is set to 1.
[0044] Since both rotation and scaling operations reduce the user's sensitivity to changes in local details, and rapid changes in either operation can produce dynamic visual masking effects, the rotation and scaling speeds are normalized separately, and the larger of the two values is taken as the dominant visual masking level in the current interaction state. When the user does not perform rotation or scaling operations, both normalized values are 0, and the perceptual masking factor is equal to 1, without changing the visual distortion term. As the interaction speed increases, the perceptual masking factor increases, reducing the weight of the visual distortion term.
[0045] The aforementioned perceptual masking factor is not used to describe the absolute physiological parameters of the human visual system, but rather to map the user interaction speed to a relative tolerance parameter in the quality level selection. The frame latency mutation ratio converts the inter-frame changes in rendering load into a unified criterion, providing actual operational data for subsequent updates of Lagrange multipliers and selection of quality levels.
[0046] S3. Update the Lagrange multipliers. Specifically, the Lagrange multipliers are updated adaptively through the feedforward and feedback channels.
[0047] See Figure 2 The figure shows the path of the feedforward channel to preset the Lagrange multiplier as the target level convergence reference value when a level switching event is detected, and the path of the feedback channel to calculate the step size amplification factor and update the Lagrange multiplier based on the frame time change ratio when no level switching event is detected. After the two paths merge, the convergence state is determined and the convergence reference value is updated.
[0048] The Lagrange multiplier represents the pixel distortion cost equivalent to a unit of rendering time, i.e., the reduction in pixel distortion gained for every 1ms increase in rendering time. The input variables for updating the Lagrange multiplier include the current state, the target state, the perceptual masking factor, the frame time abrupt change ratio, the actual rendering time of the current frame, the target frame time, the convergence reference value for each state, and the current Lagrange multiplier. The demonstration program selects feedforward or feedback processing based on whether a state switching event occurs and outputs the updated Lagrange multiplier. The Lagrange multiplier is non-negative and is used to adjust the trade-off between visual distortion and rendering time cost. Increasing the Lagrange multiplier increases the penalty for rendering time cost in the quality level selection process.
[0049] The target frame time is determined based on the mobile terminal's display refresh rate. When the display refresh rate is 30fps, the target frame time is 33.33ms. When the display refresh rate is 60fps, the target frame time is 16.67ms. When the display refresh rate is 120fps, the target frame time is 8.33ms. If the display program cannot obtain the display refresh rate, 16.67ms is used as the target frame time. The target frame time ranges from 8.33ms to 33.33ms, with the specific value determined based on the actual display refresh rate of the mobile terminal.
[0050] The demonstration program saves the Lagrange multipliers, convergence reference values, convergence status, bias correction coefficient sequence, and cumulative bias correction for each state. Data is saved using the state number as an index; when switching states, only the data corresponding to the target state is read, avoiding the direct application of historical convergence results from one state to another. Before historical convergence records are generated for each state, initial default values are used as the convergence reference values and Lagrange multipliers for that state.
[0051] The initial default value is determined based on the rendering time cost of the highest quality level for the corresponding order and the target frame time. The rendering time cost of the highest quality level is divided by the target frame time to obtain the time ratio. When the time ratio is not greater than 1, the initial Lagrange multiplier is 0. When the time ratio is greater than 1, the time ratio is subtracted by 1 and then multiplied by the base step size to obtain the initial Lagrange multiplier. The specific value of the base step size is determined through stability tests under fixed order and fixed interaction scenarios. The value that allows the Lagrange multiplier to reach a stable state within 3 to 5 frames is used as the base step size, ranging from 0.5 to 2, with 1 being preferred. By running fixed order and fixed interaction scenarios on the target mobile terminal, the number of frames required for the Lagrange multiplier to reach a stable state under different base step sizes is recorded. The value that allows stability within 3 to 5 frames is used as the base step size.
[0052] In an optional embodiment, when a state transition event is detected, the feedforward channel presets the Lagrange multipliers as convergence reference values for the target state. The convergence reference values are the weighted average of historical convergence values of the Lagrange multipliers at the target state. The feedforward channel reads the convergence reference value corresponding to the target state and presets the current Lagrange multipliers as the convergence reference value. After presetting, the preset Lagrange multipliers for the target state are output for use in selecting the quality level for the current frame. When historical convergence values already exist for the target state, the convergence reference value for the target state is read directly. When historical convergence values do not exist for the target state, an initial default value is first generated based on the ratio of the rendering time cost of the highest quality level to the target frame time, and then this initial default value is used as the convergence reference value for the target state. This preset branch can cover the case of the target state being displayed for the first time, avoiding invalid preset results due to a lack of historical data.
[0053] To continuously optimize the preset accuracy during state transitions, this invention sets the update method for the convergence reference value as follows: each time the Lagrange multiplier confirms convergence at a certain state, the Lagrange multiplier value at the time of convergence is taken as the current convergence value. The new convergence reference value is equal to the old convergence reference value multiplied by the retention coefficient plus the current convergence value multiplied by the update coefficient, with the sum of the retention coefficient and the update coefficient being 1. The retention coefficient is preferably 0.7, and the update coefficient is preferably 0.3, with both the retention coefficient and the update coefficient ranging from 0 to 1. Device performance may drift slowly due to factors such as increased operating time and rising temperature. More recent data is given higher weight in the weighting algorithm, allowing the convergence reference value to track this drift trend. The retention coefficient and the update coefficient are determined based on the rate of change of the mobile terminal's graphics processing performance; the retention coefficient is increased when device performance changes slowly, and the update coefficient is increased when device temperature or load changes rapidly. On a mobile terminal, a fixed-step model was continuously switched 50 times. Retention coefficients were set to 0.5, 0.7, and 0.9 respectively. The number of frames required for the Lagrange multiplier to reach a stable state and the steady-state error were statistically analyzed, as follows: With a retention coefficient of 0.5 and an update coefficient of 0.5, the average number of convergence frames was 8, with a relatively large steady-state error; with a retention coefficient of 0.7 and an update coefficient of 0.3, the average number of convergence frames was 4, with a relatively small steady-state error; with a retention coefficient of 0.9 and an update coefficient of 0.1, the average number of convergence frames was 12, with a slow steady-state error response. Therefore, a retention coefficient of 0.7 was selected as the preferred value. A weighted moving average method was used to ensure that recently confirmed convergence values have a high influence, while retaining historical information formed over multiple previous running cycles. The convergence criterion was that the frame time abrupt change ratio of 10 consecutive frames all fell within the range of 1 minus the normal fluctuation range to 1 plus the normal fluctuation range. At 60 frames per second, 10 frames correspond to approximately 167 milliseconds. This duration is sufficient to exclude short-term fluctuations caused by GPU scheduling jitter (typically lasting 2 to 3 frames) and is shorter than the typical duration of user operation intervals. The consecutive frame count of 10 frames can be calibrated by continuously running a fixed scene on the device and observing the minimum number of frames required for the frame latency abrupt change to enter a stable range.
[0054] When the system first runs, there are no historical convergence records for each state. At this time, the convergence reference value is taken as the initial default value. The method for setting the initial default value is: to estimate it based on the ratio of the reference time of the highest quality level of each state to the target frame time in S1. If the reference time of the highest quality level of a certain state is K times the target frame time, then the initial Lagrange multiplier is K minus 1 multiplied by the base step size of 1. Taking a certain Bai Ze state as an example, the highest level reference time is 14.2ms, the target frame time is 16.67ms, K equals 14.2 divided by 16.67 equals 0.85, K minus 1 is a negative value, which indicates that the device performance is sufficient to support the highest quality, and the initial Lagrange multiplier is 0. If it is a low-performance device, the highest level time is 33ms, K is about 2, then the initial Lagrange multiplier is 1 multiplied by the base step size, which is 1, so that the system tends to select the quality level where the reference rendering time does not exceed the target frame time in the first frame.
[0055] To eliminate the inherent deviation between the preset value and the actual convergence value and further improve the accuracy of the feedforward preset, the feedforward channel also includes deviation correction: After each step switch presets the Lagrange multiplier as the convergence reference value, the difference between the Lagrange multiplier value at the time of initial convergence confirmation and the convergence reference value is recorded and divided by the convergence reference value to obtain the deviation correction coefficient. A positive deviation correction coefficient indicates that the preset value is lower than the actual convergence value, and a negative deviation correction coefficient indicates that the preset value is higher than the actual convergence value.
[0056] When the preset convergence reference value is 0, division is not performed. The current state transition is marked as an invalid deviation correction sample, and the system waits for the target state to form a non-zero convergence reference value before re-recording. When the deviation correction coefficient is equal to 0, it is not included in the positive or negative same-sign sequence, and the current same-sign count is cleared. When the deviation correction coefficients obtained from consecutive state transitions of a preset number are all of the same sign, the product of the average of the preset number of deviation correction coefficients and the current convergence reference value is added to the convergence reference value. When the deviation correction coefficient shows alternating positive and negative values, the current same-sign sequence is cleared and the count is restarted. The preset number of consecutive same-sign occurrences is determined by repeatedly performing state transitions and recording the deviation correction coefficient sequence on the target mobile terminal. When the number of consecutive same-sign occurrences is too low and corrections are frequently triggered, the preset number is increased; when changes in device performance cannot be tracked in a timely manner, the preset number is decreased. Two consecutive occurrences of the same sign result in frequent average corrections, a fast response, and a high number of oscillations. Three consecutive occurrences of the same sign result in a moderate number of average corrections, balancing response and stability, and the lowest number of oscillations. Five consecutive occurrences of the same sign result in a slow average correction response and a reduced number of oscillations. Therefore, the optimal preset number of consecutive occurrences of the same sign is three, with a range of two to five.
[0057] The cumulative amount accumulated does not exceed a preset percentage of the convergence reference value. A preset percentage of 20% results in slow convergence, zero overshoot, and insufficient tracking speed; a preset percentage of 30% provides optimal convergence with zero overshoot; a preset percentage of 40% results in fast convergence but increased overshoot, posing a risk of overshoot. Therefore, a preset percentage of 30% is preferred, with a range of 20% to 40%. When the current correction pushes the cumulative correction amount to more than 30% of the convergence reference value, the correction amount is limited to the allowable range, and no further increase is made beyond that. Negative corrections use the same magnitude limit. The cumulative deviation correction amount is stored separately based on the initial convergence reference value for each state. When the corrected convergence reference value is less than 0, it is set to 0, and the non-negative value is used as the preset value for the next state transition.
[0058] In an optional embodiment, the feedback channel calculates a step size amplification factor based on the frame latency mutation ratio, and adjusts the update step size of the Lagrange multiplier using the step size amplification factor. Specifically, the Lagrange multiplier of the next frame is equal to the value of the current frame plus the base step size multiplied by the step size amplification factor multiplied by the normalized frame latency deviation. The normalized frame latency deviation is the actual rendering time of the current frame minus the target frame time, then divided by the target frame time.
[0059] The step size amplification factor is positively correlated with the degree to which the frame time abrupt change ratio deviates from 1. When no phase transition event is detected, the feedback channel updates the Lagrange multipliers based on the actual rendering time of the current frame, the target frame time, and the step size amplification factor. The frame time deviation is the actual rendering time of the current frame minus the target frame time, and then divided by the target frame time. The frame time deviation is positive when the actual rendering time of the current frame is greater than the target frame time; it is negative when the actual rendering time of the current frame is less than the target frame time; and it is zero when they are equal.
[0060] The step size amplification factor is calculated as follows: take the absolute value of the frame time change ratio minus 1, divide it by the normal fluctuation range, and then add 1 to obtain the step size amplification factor.
[0061] The normal fluctuation range is determined as follows: In a frame sequence where the frame latency mutation ratios of multiple consecutive frames are all within a preset stable interval, the standard deviation of the frame latency mutation ratios is calculated, and this standard deviation is used as the normal fluctuation range. The preset stable interval is centered at 1, preferably within an interval where the absolute value of the difference between the frame latency mutation ratio and 1 is no greater than 0.1. The sample standard deviation method is used to statistically analyze the frame latency mutation ratios in the stable frame sequence. The statistical range only includes frame latency mutation ratios that have passed the preset stable interval, excluding data from phase transitions or other mutated frames. The program initializes with a preset default value of 0.1 as the normal fluctuation range. After running for 30 to 100 frames, preferably 50 frames, the measured standard deviation is used to replace the preset default value. When the measured standard deviation is 0, the preset default value of 0.1 is continued until a valid standard deviation greater than 0 is obtained.
[0062] Based on the nonlinear response strategy for error mutations in control theory, fixed proportional-integral-derivative (PID) adjustment is generally used in this field. However, fixed adjustment parameters are difficult to cope with the sudden changes in rendering time when multiple tasks are performed in parallel on mobile terminals. This invention, therefore, uses the degree to which the frame time mutation ratio deviates from the normal fluctuation range as the basis for step size amplification, and constructs the following expression for the step size amplification coefficient: ; in, This is the step size amplification factor; The frame latency abrupt change ratio; This represents the normal fluctuation range, specifically the standard deviation of the frame time abrupt change ratio in a stable frame sequence. When the frame time abrupt change ratio equals 1, the step size amplification factor equals 1. As the deviation of the frame time abrupt change ratio from 1 increases, the step size amplification factor increases accordingly. When the standard deviation is less than or equal to 0, or when the standard deviation is not initialized, the standard deviation is replaced with a preset default value of 0.1 to prevent system anomalies caused by a zero denominator in division operations.
[0063] A preset upper limit is set for the step size amplification factor. When the calculated value exceeds the preset upper limit, the preset upper limit is used. The preferred preset upper limit is 5, with a range of 3 to 8. The preset upper limit is determined by simulating state change conditions such as step switching, rapid rotation, and continuous scaling on the target mobile terminal, comparing the number of convergent frames and oscillation frames under different upper limits, and selecting the minimum upper limit value that can shorten the convergence time without generating continuous oscillations. When the step size is amplified by 5 times, the Lagrange multiplier can correct the deviation within the normal change range within 2 to 3 frames. A larger amplification factor will not speed up convergence but will increase the risk of overshoot oscillation. When a state change occurs, the frame time change ratio deviates from 1, the step size amplification factor is greater than 1 but not more than 5, the step size is moderately amplified, and the Lagrange multiplier adjusts quickly to the new equilibrium point without overshoot. The GPU scheduling jitter amplitude varies across different devices. High-performance devices have small frame time fluctuations, while low-end devices have larger fluctuations. Using their respective normal fluctuation ranges as a reference makes the step size amplification factor device-independent. On devices with large fluctuations, the same deviation will not trigger excessive step size amplification.
[0064] After determining the rules for the step size amplification factor, the iterative update calculation of the Lagrange multiplier can be completed based on this factor. Specifically, the increment of the Lagrange multiplier is the product of the base step size, the step size amplification factor, and the frame time deviation. Adding the increment to the Lagrange multiplier yields the updated Lagrange multiplier, and the lower bound of the updated Lagrange multiplier is zero. To ensure that quality adjustment can quickly respond to frame rate drops on resource-constrained devices and smoothly revert when performance is sufficient, the updated Lagrange multiplier relationship is constructed by combining the multiplier update rule in rate-distortion optimization theory: ; in, For the updated Lagrange multipliers; These are the Lagrange multipliers before this update; Base step size; This is the step size amplification factor; This represents the frame time offset.
[0065] The optimal base step size is 1, based on the premise that the system can eliminate deviations within 3 to 5 frames under normal frame rate fluctuations. In actual deployment, a standard scenario can be run on the target device to observe the number of convergence frames of the Lagrange multipliers under different step size values, and a step size value that ensures the number of convergence frames is within the range of 3 to 5 frames can be selected.
[0066] Frame time deviation equals the actual rendering time of the current frame minus the target frame time, then divided by the target frame time. When the frame time deviation is positive, the increment is positive, and the updated Lagrange multiplier increases. When the frame time deviation is negative, the increment is negative, and the updated Lagrange multiplier decreases. When the value calculated by the updated Lagrange multiplier is less than 0, it is set to 0. The Lagrange multiplier is the penalty weight for rendering time cost; a negative value means that the greater the rendering time, the lower the cost, which contradicts the physical meaning. When the updated Lagrange multiplier is 0, the rendering cost term in the subsequent cost function does not produce additional penalty, and is completely dominated by the perceptual distortion term. The system selects the highest quality level, which corresponds to the situation where the device performance is extremely sufficient and is a reasonable boundary state.
[0067] When the absolute value of the difference between the frame latency change ratio and 1 for each frame within a consecutive preset number of frames is less than a preset fluctuation threshold, the Lagrange multiplier is determined to be in a convergent state. The preset fluctuation threshold is preferably 0.1, with a range of 0.05 to 0.2. The preset number of frames is preferably 10, with a range of 5 to 20 frames. At a display refresh rate of 60fps, 10 consecutive frames correspond to approximately 166.7ms, which can eliminate short-term graphics processing jitter. If the absolute value of any frame is greater than or equal to the preset fluctuation threshold, the current sequence is determined not to be convergent, and the consecutive frame counting restarts. When the Lagrange multiplier is determined to be in a convergent state, the Lagrange multiplier value of the current frame is recorded as the current convergent value, and the convergence reference value update for the corresponding state is triggered. If the current frame is also at the first convergence position after a state transition, the deviation correction coefficient is first calculated using the Lagrange multiplier value, and then the convergence reference value is updated according to the retained coefficient and the updated coefficient.
[0068] The feedforward channel leverages the predictable changes in model complexity during state transitions in the ancient mythical beast model, skipping the frame-by-frame trial-and-error process and allowing the Lagrange multipliers to directly reach the vicinity of the historical operating values of the target state. The feedback channel amplifies the adjustment step size to accelerate response when the rendering load of the ancient mythical beast model changes, while maintaining small adjustments to ensure stability during smooth operation. Convergence reference values and deviation corrections are continuously updated during operation, improving feedforward accuracy according to the mobile terminal's operating state. The improved adaptive step-size feedback algorithm and feedforward preset mechanism are closely integrated with the dynamic interactive scenarios of the ancient mythical beast model, solving the problem of computational power imbalance during viewpoint changes and form transitions in complex models.
[0069] S4. Solve for the optimal quality level of the current frame. Specifically, based on the Lagrange multiplier, perceptual masking factor, rendering time cost, and visual distortion, solve for the optimal quality level of the current frame using the rate-distortion cost function.
[0070] The rendering program receives multiple quality levels corresponding to the current state, the rendering time cost and visual distortion of each quality level, and the perceptual masking factor and updated Lagrange multipliers of the current frame. Multiple quality levels correspond to the current state number and the target operating device; measurement data from other states or other target operating devices cannot be directly used for the current frame calculation. Cost calculation proceeds only when the input data is complete and consists entirely of valid non-negative numbers. If any quality level lacks rendering time cost or visual distortion, the corresponding quality level is marked as unselectable and not included in the minimum cost comparison for the current frame.
[0071] In an optional embodiment, the rate-distortion cost function calculates the cost for each quality level. The cost is the sum of the perceptual distortion term and the rendering cost term. The perceptual distortion term is the visual distortion of the quality level divided by the perceptual masking factor. The rendering cost term is the product of the updated Lagrange multiplier and the rendering time cost of the corresponding quality level. The optimal quality level is the one with the lowest cost among all quality levels. This cost calculation borrows from the standard rate-distortion Lagrange cost function form in the video coding field, representing quality loss in terms of visual distortion and resource consumption in terms of rendering time cost.
[0072] The input to the perceptual distortion term is the visual distortion degree and the perceptual masking factor. The processing rule is to divide the visual distortion degree by the perceptual masking factor, and output the effective perceptual distortion term under the current operating state, representing the effective perceptual loss of this quality level under the current user operating state, with the dimension in pixels. The larger the perceptual masking factor, the smaller the effective perceptual distortion term for the same quality level, indicating that the user's sensitivity to detail differences is reduced under fast interaction conditions. The input to the rendering cost term is the updated Lagrange multiplier and the rendering time cost. The processing rule is to multiply the updated Lagrange multiplier by the rendering time cost, and output the rendering cost term corresponding to the current quality level, representing the degree to which the selected quality level occupies the frame time budget, with the dimension also in pixels. When the updated Lagrange multiplier increases, the rendering cost term corresponding to the quality level with longer rendering time increases more, and the quality level selection tends to favor the quality level with shorter rendering time.
[0073] The cost calculation for each quality level uses the same perceptual masking factor and Lagrange multipliers. The rendering program sequentially reads the corresponding visual distortion and rendering time cost according to the quality level number, first calculates the perceptual distortion term, then calculates the rendering cost term, and finally adds the two together to obtain the cost of the corresponding quality level. This invention draws on the standard rate-distortion Lagrange cost function form in the field of video coding and improves it by combining rotational angular velocity and scaling rate. It uses visual distortion to represent quality loss and rendering time cost to represent resource consumption. The perceptual masking factor is introduced into the denominator to quantify the blurring effect of the human eye during dynamic operation, constructing the cost value. After all selectable quality levels have been calculated, the quality level with the smallest cost value is selected as the optimal quality level for the current frame.
[0074] Taking a state of the mythical beast Bai Ze from ancient texts as an example, the visual distortion at the four quality levels is 2.8px, 1.2px, 0.4px, and 0px, respectively, with rendering times of 3ms, 5.1ms, 8.6ms, and 14.2ms. The perceptual masking factor for the current frame is 1.5, derived from the perceptual masking calculation result at the corresponding interaction speed. The Lagrange multiplier is 3.2, derived from the current Lagrange multiplier obtained iteratively through the feedback channel during the preceding execution of this device and this state. The cost of quality level 1 is: (2.8 / 1.5) + (3.2 × 3) = 1.87 + 9.6 = 11.47; the cost of quality level 2 is: (1.2 / 1.5) + (3.2 × 5.1) = 0.8 + 16.32 = 17.12; the cost of quality level 3 is: (0.4 / 1.5) + (3.2 × 8.6) = 0.27 + 27.52 = 27.79; the cost of quality level 4 is: (0 / 1.5) + (3.2 × 14.2) = 0 + 45.44 = 45.44. Since quality level 1 has the lowest value, the system outputs quality level 1 as the optimal quality level for the current frame.
[0075] When the user stops rotating and scaling the model, the perceptual masking factor returns to 1. At this point, the cost of each quality level is recalculated using the updated Lagrange multipliers of the current frame, the rendering time cost of the current order, and the visual distortion. The optimal quality level before the interaction state change cannot be directly used.
[0076] When the cost values are the same, a deterministic parallel processing rule is adopted. If multiple quality levels have the same cost value, the quality level with the lower number is selected as the optimal quality level; if there is only one selectable quality level, the corresponding quality level is directly output as the optimal quality level; if all quality levels in the current state lack valid cost data, the quality level actually executed in the previous frame is retained as a temporary output; when the program runs for the first time and there is no quality level actually executed in the previous frame, quality level 1 is output.
[0077] The cost function allows for comparison of different quality levels under a unified standard. User operation speed and mobile terminal rendering load both participate in the selection process, avoiding the frequent oscillations and performance mismatch issues that occur with fixed threshold strategies when the load fluctuates.
[0078] S5. Apply inter-frame smoothing constraints and perform rendering. Specifically, apply inter-frame smoothing constraints to the optimal quality level, perform rendering, and feed back the actual rendering time to the next frame.
[0079] The inter-frame smoothing process receives the optimal quality level of the current frame, the quality level actually executed in the previous frame, the preset maximum jump amplitude, and the number of quality levels corresponding to the current state. It uses the optimal quality level of the current frame as the target level, the quality level actually executed in the previous frame as the current level, and determines whether the absolute value of the difference between the target level and the current level exceeds the preset maximum jump amplitude, thus outputting the execution level of the current frame.
[0080] In an optional embodiment, the inter-frame smoothing constraint is implemented as follows: Let the optimal quality level of the current frame be the target level, and the quality level actually executed in the previous frame be the current level. If the absolute value of the difference between the target level and the current level exceeds a preset maximum jump amplitude, then the execution level of the current frame is restricted to a preset maximum jump amplitude along the target level direction. When the target level is greater than the current level and the difference between the target level and the current level exceeds the preset maximum jump amplitude, the current level is increased by the preset maximum jump amplitude and used as the execution level. When the target level is less than the current level and the difference between the target level and the current level exceeds the preset maximum jump amplitude, the current level is decreased by the preset maximum jump amplitude and used as the execution level. When the absolute value of the difference between the target level and the current level does not exceed the preset maximum jump amplitude, the target level is directly determined as the execution level.
[0081] The preset maximum jump amplitude is preferably set to one quality level, with a value ranging from one to two quality levels. Adjacent quality levels are established by progressively reducing model complexity. A single change of one quality level can reduce visual flicker caused by sudden changes in image quality. The specific value is determined based on user visual evaluation results, and is calculated by subtracting one from the minimum number of jump levels that the test user can stably perceive as a sudden change in image quality.
[0082] The execution level value also needs to undergo a range check. If the calculated execution level is less than 1, the execution level is corrected to 1. If the calculated execution level is greater than the number of quality levels for the current stage, the execution level is corrected to the highest quality level for the current stage. If the target level is invalid but the current level is valid, the current level is retained. If both the target level and the current level are invalid, the execution level is set to quality level 1.
[0083] When a state transition event occurs, the current quality level is reset to the quality level obtained by solving the rate-distortion cost function in the new state. The reset quality level is calculated using the Lagrange multipliers preset in the feedforward channel during the state transition, the current perceptual masking factor, and the rendering time cost and visual distortion of the new state. After the reset, the reset quality level is used as the current level of the new state, and inter-frame smoothing constraints are executed from the next frame according to the preset maximum jump amplitude.
[0084] After determining the execution level, the rendering program reads the mesh data, material data, and texture data corresponding to the current stage and execution level. It then submits these data to the mobile terminal's graphics rendering interface, completes model rendering under the current frame's viewpoint, lighting, and display resolution, and outputs the current frame's image. If the data corresponding to the execution level fails to load, the program searches for a loadable quality level in the current stage in descending order of quality level, and uses the first loadable quality level as the temporary execution level.
[0085] At the start of rendering, the start timestamp of the current frame is recorded; upon completion of rendering and submission of display data, the end timestamp is recorded. The actual rendering time of the current frame is obtained by subtracting the start timestamp from the end timestamp, and the actual rendering time, the actual quality level of the current frame, and the current state number are written into the runtime data. If the actual rendering time of the current frame is valid, it is used as the actual rendering time of the previous frame for calculating the frame time jump ratio of the next frame. If the actual rendering time of the current frame is invalid or not greater than 0, the previous valid time record is not overwritten, and the frame time jump ratio of the next frame is treated as a no-jump state. The execution level of the current frame is saved as the current level of the next frame after rendering is completed.
[0086] With the range of quality level changes between frames constrained, users will not perceive sudden jumps in image quality during continuous rotation or scaling of the model. When switching between different frame states, a new quality level benchmark is established within the new state, preventing model differences between old and new states from being incorrectly mapped as abrupt changes in inter-frame quality. Actual rendering time is fed back to subsequent frames, allowing the quality level selection to be continuously adjusted based on the rendering results, forming a complete feedback and scheduling system.
[0087] To verify the effectiveness of the adaptive rendering scheduling strategy of this invention, a control experiment was set up. The experiment used the same 3D ancient book model, a unified mobile terminal hardware environment, and a completely consistent sequence of interactive operations. Rendering data for 100 consecutive frames was continuously collected, and the changes in rendering time per frame were statistically analyzed. The control group used an existing fixed rendering precision strategy: rendering the 3D model with constant mesh precision throughout. The experimental group used the adaptive rendering strategy of this invention: real-time acquisition of user rotation and scaling interaction information to calculate the perceptual masking factor, relying on a dual-channel Lagrange multiplier update mechanism, enabling the feedforward channel to preset convergence reference values during state switching, dynamically amplifying the update step size through the feedback channel in scenarios with sudden changes in rendering load, and adaptively selecting the optimal model precision level by combining inter-frame smoothing constraints.
[0088] Experimental results are as follows Figure 4 As shown, the fixed-precision strategy in the control group, during the user's rapid model rotation and model phase switching, consistently uses a fixed quality level for rendering, failing to adjust model complexity based on the current interaction state and device load changes. When frames 20-40 enter the rapid rotation phase, changes in the visible area increase the computational load of model rendering, resulting in a significant increase in single-frame rendering time. When phase switching occurs between frames 60-80, short-term load peaks arise due to model data updates and rendering resource reorganization, further increasing rendering time. In contrast, the method of this invention dynamically adjusts the quality level based on real-time interaction state and device performance, keeping rendering time within the target frame time range and reducing frame rate fluctuations. Furthermore, in scenarios with sudden load changes, such as rapid user model rotation and model state switching, the control group's fixed-precision strategy resulted in an average frame time of 22ms, with a sharp increase in single-frame rendering time exceeding the target frame time of 16.67ms, a frame time standard deviation of 5.6ms, and 12 stutters, exhibiting significant frame rate jitter and screen stuttering. In contrast, the experimental group of this invention, under the same sudden load conditions, achieved an average frame time of 16ms, a frame time standard deviation of 1.8ms, and 3 stutters, with only a small, instantaneous increase in rendering time. It quickly converged to the target frame time range, with a smaller fluctuation in the rendering time curve. Compared to existing fixed-precision rendering solutions, this invention can dynamically adapt to interactive scenarios and sudden changes in rendering load, suppressing drastic fluctuations in single-frame rendering time, and continuously maintaining a stable target frame rate. While controlling visual distortion, it also alleviates stuttering issues during 3D model browsing.
[0089] This invention also discloses an immersive display system for ancient book culture, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, an immersive display method for ancient book culture according to this invention is implemented.
[0090] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for immersive display of ancient book culture, characterized in that, include: S1. Establish multiple quality levels for each stage model and obtain the rendering time cost and visual distortion of each quality level; S2. Before rendering each frame, collect the user interaction status, calculate the perceptual masking factor, and calculate the ratio of the actual rendering time of the current frame to that of the previous frame to obtain the frame time change ratio. S3. The Lagrange multipliers are adaptively updated through the feedforward and feedback channels. When a state transition event is detected, the feedforward channel presets the Lagrange multipliers to the convergence reference value of the target state. The convergence reference value is the weighted average of the historical convergence values of the Lagrange multipliers in that state. The feedback channel calculates the step size amplification factor based on the frame time abrupt change ratio, and adjusts the update step size of the Lagrange multiplier with the step size amplification factor. The step size amplification factor is positively correlated with the degree to which the frame time abrupt change ratio deviates from 1. S4. Based on the updated Lagrange multipliers, perceptual masking factor, rendering time cost, and visual distortion, solve for the optimal quality level of the current frame using the rate-distortion cost function. S5. After applying inter-frame smoothing constraints to the optimal quality level, render the frame and feed back the actual rendering time to the next frame.
2. The immersive display method for ancient book culture according to claim 1, characterized in that, The process of collecting user interaction status and calculating perceptual masking factors before rendering each frame includes: Collect user interaction data including rotational angular velocity and zoom rate; The rotation normalization value is obtained by dividing the absolute value of the rotation angular velocity by the angular velocity reference value, and the scaling normalization value is obtained by dividing the absolute value of the scaling rate by the scaling rate reference value. The larger of the rotation normalization value and the scaling normalization value is then added to 1 to obtain the perception masking factor.
3. The immersive display method for ancient book culture according to claim 1, characterized in that, The method for updating the convergence reference value is as follows: each time the Lagrange multiplier is confirmed to converge in a certain order state, the Lagrange multiplier value at the time of convergence is taken as the current convergence value; the new convergence reference value is equal to the old convergence reference value multiplied by the retention coefficient plus the current convergence value multiplied by the update coefficient; The sum of the retention coefficient and the update coefficient is 1. When the absolute value of the difference between the frame time change ratio of each frame within a consecutive preset number of frames and 1 is less than the preset fluctuation threshold, the Lagrange multiplier is determined to be in a convergent state.
4. The immersive display method for ancient book culture according to claim 1, characterized in that, The feedforward channel includes deviation correction: after each step state switch, the Lagrange multiplier is preset as the convergence reference value, and the difference between the Lagrange multiplier value at the time of the first confirmed convergence and the convergence reference value is divided by the convergence reference value to obtain the deviation correction coefficient. When the deviation correction coefficients obtained from a preset number of consecutive state switchings all have the same sign, the product of the average of the preset number of deviation correction coefficients and the current convergence reference value is added to the convergence reference value; wherein, the cumulative amount of the addition is less than or equal to a preset proportion of the convergence reference value, and the corrected convergence reference value is 0 when it is less than 0.
5. The immersive display method for ancient book culture according to claim 1, characterized in that, The adjustment of the update step size of the Lagrange multipliers by the step size amplification factor includes: The increment of the Lagrange multiplier is obtained by multiplying the base step size, the step size amplification factor, and the frame time deviation. Adding the increment of the Lagrange multiplier to the Lagrange multiplier yields the updated Lagrange multiplier; The lower bound of the updated Lagrange multiplier is 0, and the frame time deviation is the actual rendering time of the current frame minus the target frame time and then divided by the target frame time.
6. The immersive display method for ancient book culture according to claim 5, characterized in that, The method for obtaining the step size amplification factor is as follows: take the absolute value of the frame time change ratio minus 1, divide it by the normal fluctuation range, and then add 1; Among them, a preset upper limit is set. When the calculated step size amplification factor exceeds the preset upper limit, the preset upper limit is taken as the step size amplification factor; otherwise, the calculated step size amplification factor is taken as the step size amplification factor.
7. The immersive display method for ancient book culture according to claim 6, characterized in that, The method for obtaining the normal fluctuation range is as follows: in a frame sequence in which the frame time change ratio of multiple consecutive frames is within a preset stable range, the standard deviation of the frame time change ratio is calculated, and the standard deviation of the frame time change ratio is taken as the normal fluctuation range. During system initialization, a preset default value is used as the normal fluctuation range. After running a preset number of frames, the preset default value is replaced with the standard deviation of the measured frame time change ratio.
8. The immersive display method for ancient book culture according to claim 1, characterized in that, The rate-distortion cost function calculates the cost value for each quality level. The cost value is the sum of the perceptual distortion term and the rendering cost term. The perceptual distortion term is the visual distortion degree of the quality level divided by the perceptual masking factor. The rendering cost term is the product of the updated Lagrange multiplier and the rendering time cost of the quality level. The quality level with the lowest cost value among all quality levels is selected as the optimal quality level. If multiple quality levels have the same cost value, the quality level with the lower number is selected as the optimal quality level.
9. The immersive display method for ancient book culture according to claim 1, characterized in that, The inter-frame smoothing constraints include: setting the optimal quality level of the current frame as the target level, and the quality level actually executed in the previous frame as the current level; when the target level is greater than the current level and the absolute value of the difference between the two exceeds the preset maximum jump amplitude, the current level is increased by the preset maximum jump amplitude and then used as the execution level; when the target level is less than the current level and the absolute value of the difference between the two exceeds the preset maximum jump amplitude, the current level is decreased by the preset maximum jump amplitude and then used as the execution level; when the absolute value of the difference between the target level and the current level does not exceed the preset maximum jump amplitude, the target level is directly determined as the execution level; when a state switching event occurs, the current level is reset to the quality level obtained by solving the rate-distortion cost function in the new state.
10. An immersive display system for ancient book culture, characterized in that, include: A processor and a memory, the memory storing computer program instructions that, when executed by the processor, implement the immersive display method for ancient book culture according to any one of claims 1-9.
Citation Information
Patent Citations
Real-time display method for cultural relic ultra-high-definition 3D scanning model
CN114332405A
