A multi-client-based digital human display and interaction system

Abstract

The application relates to the technical field of digital people and discloses a digital person display and interaction system based on multiple clients, which comprises a server system and a heterogeneous client terminal; the heterogeneous client terminal collects runtime hardware state data to generate terminal capability scores, captures user input and converts the same into virtual focus data which is sent to the server; the server system comprises a resource arrangement module and a gradient difference coding module; the resource arrangement module calculates the current computing power budget connected according to the terminal capability scores; the gradient difference coding module divides original video frames into different picture quality weight regions according to the virtual focus data, and generates compressed video streams with non-uniform quality distribution in combination with the computing power budget. Through an end-cloud collaborative mechanism, the application realizes adaptive resource scheduling and differential coding based on terminal capability and interaction intention, reduces the bandwidth and computing power load, and guarantees the interaction experience on the heterogeneous devices.

Description

Technical Field

[0001] This invention relates to the field of digital human technology, specifically to a multi-client-based digital human display and interaction system. Background Technology

[0002] With the development of virtual reality and artificial intelligence, high-fidelity digital humans are increasingly being used in scenarios such as financial services, virtual tour guides, and online education. To overcome the hardware performance bottlenecks of terminal devices, current mainstream solutions mostly employ cloud rendering technology. This involves real-time rendering and video encoding of the digital human on the server side, and then transmitting the video footage to the client for decoding and display via streaming media technology. This approach shifts the graphics computing burden from the device side to the cloud side, enabling even lightweight terminals to display high-quality virtual avatars.

[0003] However, existing streaming media transmission solutions primarily rely on adaptive bitrate algorithms based on network bandwidth, dynamically adjusting the video bitrate according to network throughput and packet loss rate. While this mechanism addresses network fluctuations, it often overlooks the video decoding capabilities and operational thermal states of the terminal device itself. When the server sends high-resolution, high-bitrate video streams to a terminal with weak decoding capabilities or in a hot-throttling state, even with sufficient network bandwidth, the terminal's hardware decoder may experience backlog due to excessive load, leading to increased latency, playback stuttering, and even overheating and frequency throttling, severely impacting the user experience.

[0004] Furthermore, digital human interaction scenarios demand high real-time performance and image detail, with users typically focusing on the digital human's face. Traditional video encoding technologies often employ full-frame uniform quantization or region-of-interest (ROI) encoding based on fixed rules, making it difficult to adapt to rapidly changing user interaction intentions. Given the inherent network transmission latency and encoding time, encoding solely based on the current interaction coordinates often results in high-quality areas lagging behind the user's actual line of sight or focus, causing visual discontinuity and blurriness. Simultaneously, existing cloud rendering architectures lack deep collaboration between the client and cloud, failing to fully utilize local client computing resources to compensate for degraded image quality, resulting in suboptimal display effects under limited bandwidth or low computing budgets. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a digital human display and interaction system based on multiple clients, which solves the problems of lag and image quality mismatch in digital human interaction experience caused by differences in hardware performance of heterogeneous terminals and network fluctuations.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: a digital human display and interaction system based on multiple clients, including a server system, a communication network, and heterogeneous client terminals.

[0007] The heterogeneous client terminal is responsible for collecting runtime hardware status data and generating terminal capability scores, while capturing user input operations and converting them into virtual focus data;

[0008] The server system includes a resource orchestration module and a gradient differential coding module. The resource orchestration module receives the terminal capability score and calculates the computing power budget for the current connection.

[0009] The gradient differential coding module receives the original video frame, divides the video frame into regions with different image quality weights according to the virtual focus data, and generates a compressed video stream with non-uniform quality distribution by combining the computing power budget.

[0010] The heterogeneous client terminal receives and decodes the compressed video stream and renders the image to the display device.

[0011] The heterogeneous client terminals periodically collect runtime hardware status data, including heap memory usage, video decoding latency, and device thermal throttling status, through a performance telemetry module. The system uses a weighted normalization model to calculate the terminal capability score based on this data. The specific calculation logic is as follows:

[0012] The ratio of heap memory utilization to the maximum heap memory limit, the ratio of video decoding latency to the target frame interval, and the device thermal throttling status indicator are all weighted and summed. When the video decoding latency exceeds the target frame interval, the terminal's capability score decreases, thus reflecting the terminal's load pressure to the server.

[0013] The heterogeneous client terminal establishes a unit Cartesian coordinate system through an interactive normalization module, projecting signals from different input modes into normalized coordinates.

[0014] When the input device is in discrete input mode, the system traverses the document object model tree of the current page, filters the focusable elements, and calculates a navigation score based on the navigation direction. This score combines orthogonal distance and projected distance weights to predict potential interaction areas.

[0015] When the input device is in continuous input mode, the system uses a kinematic trajectory extrapolation algorithm to maintain a circular queue of historical sampling points and calculate a smooth velocity vector. It combines the round-trip network latency and the average encoding time of the server to determine the look-ahead time window, and then calculates the predicted target coordinates to compensate for the latency caused by signal transmission and video encoding.

[0016] The resource orchestration module uses a nonlinear mapping function to map the terminal capability score to an instantaneous budget value, and controls the decay rate of the instantaneous budget value as the terminal capability score decreases through a computing power sensitivity index. Simultaneously, the system performs an exponentially weighted moving average on the instantaneous budget value to generate the final effective computing power budget. This computing power budget serves as a constraint threshold, limiting the upper limit of computational load for macroblock processing during subsequent video encoding.

[0017] The gradient differential coding module spatially divides the original video frame into a core high-definition region, a prediction transition region, and a background suppression region based on the virtual focus data. The core high-definition region is geometrically centered on the virtual focus coordinates; the prediction transition region is formed by subtracting the core high-definition region from the extended region generated by the prediction points; and the background suppression region is the remaining part of the video frame.

[0018] The system assigns incremental base quantization parameters to the core high-quality region, the predicted transition region, and the background suppression region. Based on this, the gradient differential coding module generates a macroblock-level target quantization parameter map: a motion-adaptive quantization parameter offset is generated by calculating the macroblock's motion vector magnitude, and a texture-adaptive quantization parameter offset is generated by calculating the macroblock's spatial texture complexity; the initial quantization parameters are then superimposed with the aforementioned offsets and clamped to obtain the final macroblock-level target quantization parameters.

[0019] The resource orchestration module also executes a dynamic coding constraint strategy based on computing power budget. The system calculates a computing power budget factor associated with the coding queue length and the coding time of the previous frame; it calculates a retention priority score based on the coding cost and transition probability of the predicted region, and filters the predicted regions according to the score to ensure that the total coding cost does not exceed the limit of the computing power budget factor; for the retained predicted regions, a region fusion operation is performed when the proportion of the merged blank region area does not exceed a preset threshold.

[0020] The media rendering container module of the heterogeneous client terminal performs visual compensation. During rendering, the system calculates the deviation distance between the actual gaze point and the predicted gaze point at the encoding end; if the deviation distance exceeds a threshold, a dynamic desharpening mask is applied to the area around the actual gaze point, with the sharpening intensity decreasing Gaussian with distance, in order to enhance image details on the display end and improve the user's subjective visual experience.

[0021] This invention provides a multi-client-based digital human display and interaction system. It has the following beneficial effects:

[0022] 1. This invention uses a performance telemetry module on heterogeneous client terminals to collect heap memory, decoding latency, and hot throttling status in real time. Combined with a server-side resource orchestration module, it maps the terminal capability score to a dynamic computing power budget. The closed-loop control mechanism of end-cloud collaboration enables the server to adaptively adjust the macroblock budget of video encoding according to the actual load capacity of the terminal, avoiding decoding stuttering and application crashes caused by insufficient terminal hardware performance or overheating, and ensuring the stable operation of the digital human system on devices with different performance levels.

[0023] 2. This invention utilizes an interactive normalization module to capture user input and predict potential interaction areas, and works with a gradient differential coding module to generate a compressed video stream with a non-uniform quality distribution. The system concentrates limited bitrate resources on the core high-quality areas of interest to the user and the predicted transition areas. While reducing transmission bandwidth usage and decoding computing power requirements, the system compensates for the system latency caused by network transmission and video encoding through a trajectory extrapolation algorithm, ensuring fast operation response and visual clarity in high-frequency interactive scenarios.

[0024] 3. This invention introduces a media rendering container module in a heterogeneous client terminal to execute a visual compensation strategy. When a large deviation is detected between the user's actual gaze point and the prediction at the encoding end, a dynamic desharpening mask is applied to the area around the gaze point. The remaining computing power on the terminal side is fully utilized to perform post-processing enhancement on the decoded image, which effectively improves the subjective visual experience in background suppression areas or low bit rate transmission scenarios, and improves the final display effect without increasing the encoding load on the server side. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0026] Figure 2 This is a flowchart illustrating the dynamic computing power budget allocation logic of the present invention.

[0027] Figure 3 This is a graph illustrating the adaptive adjustment of computing power budget based on terminal status awareness according to the present invention.

[0028] Among them, 100 is the server system; 110 is the digital human rendering engine module; 120 is the streaming media service module; 130 is the signaling control module; 140 is the resource orchestration module; 150 is the gradient differential coding module; 200 is the heterogeneous client terminal; 210 is the unified web kernel module; 220 is the performance telemetry module; 230 is the interaction normalization module; 240 is the media rendering container module; 300 is the communication network; 310 is the media data channel; and 320 is the signaling interaction channel. Detailed Implementation

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] See attached document Figure 1 This invention provides a multi-client-based digital human display and interaction system, which includes a server system 100, a communication network 300, and heterogeneous client terminals 200. The server system 100 and the heterogeneous client terminals 200 establish a connection through the communication network 300 to perform bidirectional data interaction and command transmission.

[0031] The server system 100 provides computing power support for digital human generation, streaming media distribution, and global session state management. The server system 100 includes a digital human rendering engine module 110, a streaming media service module 120, a signaling control module 130, a resource orchestration module 140, and a gradient differential coding module 150.

[0032] The digital human rendering engine module 110 is used to generate raw audio and video data streams containing digital human image, movements, and voice. The digital human rendering engine module 110 receives drive commands from the signaling control module 130 and synthesizes the digital human's facial expressions, lip-sync animation, and body movements in real time.

[0033] The streaming media service module 120 is built on the WebRTC protocol and is responsible for establishing and maintaining real-time media transmission sessions between the server system 100 and heterogeneous client terminals 200. The streaming media service module 120 performs media stream encapsulation, transmission control protocol processing, and packet loss retransmission control.

[0034] The signaling control module 130 maintains a persistent connection with the heterogeneous client terminal 200 via a WebSocket or WebRTC data channel. The signaling control module 130 is responsible for handling session establishment requests, receiving status data reported by the client, issuing control commands, and synchronizing the service status of multiple terminals.

[0035] Resource orchestration module 140 is connected to signaling control module 130. Resource orchestration module 140 receives performance feedback data from heterogeneous client terminals 200 and calculates the computing power budget and bandwidth quota for the current connection. Based on the real-time processing capabilities of the terminals, resource orchestration module 140 dynamically generates encoding constraint strategies for the connection.

[0036] The gradient differential encoding module 150 is located between the digital human rendering engine module 110 and the streaming media service module 120. The gradient differential encoding module 150 receives raw video frames and, based on the constraint strategy output by the resource orchestration module 140 and the interactive focus data reported by the heterogeneous client terminal 200, generates a compressed video stream with a non-uniform quality distribution. The gradient differential encoding module 150 applies differentiated quantization parameters to different regions within the video frame.

[0037] The communication network 300 includes an independent media data channel 310 and a signaling interaction channel 320. The media data channel 310 is used to transmit audio and video streams compressed by the gradient differential coding module 150. The signaling interaction channel 320 is used to transmit non-media structured data, including performance telemetry data, interactive coordinate data, session state synchronization data, and control signaling.

[0038] The heterogeneous client terminal 200 is a front-end device for user interaction, including desktop computers, mobile terminal devices, and smart TV terminals. The heterogeneous client terminal 200 includes a unified Web kernel module 210, a performance telemetry module 220, an interaction normalization module 230, and a media rendering container module 240.

[0039] The unified Web kernel module 210 runs on the operating system of the heterogeneous client terminal 200, and is responsible for loading and executing standardized business logic code, interface layout rendering, and state management. The unified Web kernel module 210 shields the differences in the underlying operating system and provides a unified runtime environment.

[0040] The performance telemetry module 220 is embedded in the underlying layer of the unified Web kernel module 210. The performance telemetry module 220 periodically collects runtime hardware status data from heterogeneous client terminals 200, including heap memory usage, video decoding latency, and device hot throttling status. The performance telemetry module 220 generates terminal capability vectors from the collected data and sends them to the resource orchestration module 140 of the server system 100 via the signaling interaction channel 320.

[0041] The interaction normalization module 230 is used to process user input operations. It converts discrete operation signals or continuous coordinate signals from different input devices into virtual focus data in a unified coordinate system. Furthermore, the interaction normalization module 230, based on the current interface topology, calculates and predicts the potential interaction area, and sends the virtual focus data and predicted area data to the server system 100 via the signaling interaction channel 320.

[0042] The media rendering container module 240 is connected to the media data channel 310. The media rendering container module 240 receives the video stream from the server system 100, calls the hardware decoder of the heterogeneous client terminal 200 to decode it, and renders the decoded image frames to the display device. The media rendering container module 240 simultaneously monitors the buffer status and frame processing time during the decoding process and feeds this measurement data back to the performance telemetry module 220.

[0043] During system operation, the performance telemetry module 220 monitors the load of the heterogeneous client terminals 200 in real time, while the interaction normalization module 230 monitors the user's areas of interest in real time. The server system 100 uses the resource orchestration module 140 to evaluate the terminal's capabilities and the gradient differential coding module 150 to allocate coding bit resources within a single frame. The video stream delivered through the media data channel 310 maintains the clarity of the core interactive areas while reducing the amount of data in non-interested areas, thus matching the decoding load of the media rendering container module 240 with the real-time capabilities of the heterogeneous client terminals 200.

[0044] The performance telemetry module 220 performs a quantitative assessment of the hardware resource status of the heterogeneous client terminal 200 during runtime. This module acquires real-time data reflecting the current processing capacity of the device by establishing a multi-dimensional observation index system, and calculates the terminal capability score based on the data, so that the server system 100 can make resource scheduling decisions.

[0045] The real-time hardware status data collected by the performance telemetry module 220 includes at least three dimensions: heap memory utilization, video decoding latency, and device thermal throttling status.

[0046] The performance telemetry module 220 obtains memory usage information through the performance interface provided by the heterogeneous client terminal 200 runtime environment. For a Chrome-based web runtime environment, the performance telemetry module 220 reads the `jsHeapSizeLimit` and `totalJSHeapSize` properties through the `performance.memory` interface. The `jsHeapSizeLimit` property represents the maximum heap memory limit allowed by the current JavaScript context, and the `totalJSHeapSize` property represents the total amount of heap memory currently allocated. Heap memory utilization rate. Defined as the current time The ratio of allocated heap memory to the maximum heap memory limit. When the standard Web interface is unavailable, such as in a partially customized WebView container, the performance telemetry module 220 calls the operating system's native layer system service interface to obtain the PSS memory value of the current application process, and injects this value into the context of the unified Web kernel module 210 through the evaluateJavascript method or the addJavascriptInterface mapping object.

[0047] When processing each frame of video data, the media rendering container module 240 records the timestamp of the frame data input to the hardware decoder. Timestamp of the output YUV data after decoding The performance telemetry module 220 calculates continuous data within a set time window. Decoding time per video frame The arithmetic mean of the values ​​is used to obtain the video decoding delay time. The video decoding latency directly reflects the load on the graphics processor or central processing unit of the heterogeneous client terminal 200 in processing the video stream at the current resolution and bitrate. When the video decoding latency exceeds the frame interval of the video stream, it indicates that the terminal's processing capacity is insufficient.

[0048] The performance telemetry module 220 obtains the current thermal throttling status by listening to the operating system's underlying thermal status callback interface. For example, in the Android system, it listens for the THERMAL_STATUS_THROTTLING signal through the PowerManager and Thermalservice services. A thermal throttling status indicator factor is defined. When the device is detected to be in normal operating condition without temperature control triggered, The value is 1; when the device triggers thermal throttling and frequency reduction, A coefficient with a value less than 1. This coefficient is mapped hierarchically based on the thermal state enumeration value returned by the system. For example, it takes a value of 0.8 under mild throttling conditions and a value of 0.5 under severe throttling conditions.

[0049] The performance telemetry module 220 inputs the collected heap memory usage, video decoding latency, and thermal throttling status indicator into a preset scoring model to calculate the current time. Terminal capability score .

[0050] Terminal capability score The calculation uses a weighted normalization model, and its formula is as follows:

[0051] ;

[0052] in: Indicates the current time The heap memory usage rate, with a value ranging from 0 to 1; Indicates the current time The video decoding latency, in milliseconds; This represents the reference decoding delay time, and its value is the target frame interval time of the video stream. It is a non-zero small constant; This is a function that takes the minimum value and limits the upper limit of the decoding latency score item to 1 to prevent score overflow in low-latency scenarios. Indicates the current time Thermal throttling state indicator factor; , , These are the weighting coefficients corresponding to heap memory utilization, video decoding latency, and hot throttling status indicator factors, respectively, and satisfy the following conditions: .

[0053] The performance telemetry module 220 executes the above data acquisition and calculation methods according to a preset cycle, and outputs the calculated terminal capability score. Encapsulated in a signaling data packet, the score is sent to the server system 100 via signaling interaction channel 320. The server system 100 dynamically adjusts the computing power budget for subsequent video encoding based on this score. Specifically, the server system 100 has a preset macroblock processing budget baseline value. and the received Mapped to macroblock processing budget via exponential decay function .when When the alarm threshold is below the preset threshold, the exponential decay function makes... It decreases at a rate faster than linear, thereby forcibly triggering subsequent ROI region pruning logic to achieve proactive protection for low-performance terminals.

[0054] See attached document Figure 2 Based on the received terminal capability score, the resource orchestration module 140 calculates the macroblock processing budget to constrain the complexity of subsequent video coding. This process establishes a mathematical mapping between the physical computing power boundaries of the heterogeneous client terminal 200 and the video stream coding parameters, ensuring that the decoding load of the video stream is always kept within the terminal's current real-time processing capabilities.

[0055] The resource orchestration module 140 calculates the macroblock throughput theoretically required to maintain full-frame playback at the current video session resolution and frame rate. A macroblock is the basic processing unit in a video coding standard, typically a 16x16 pixel block. Macroblock processing budget baseline. This represents the total number of macroblocks that the terminal can stably decode per second under ideal hardware conditions. The calculation formula is:

[0056] ;

[0057] in: Indicates the target width of the video stream; Indicates the target height of the video stream; Indicates the target frame rate of the video stream; This indicates the rounding up operation; This represents the scaling factor for the encoding standard complexity. This is used to balance the differences in decoding complexity among different video coding standards. For example, using the H.264 standard as a benchmark, it sets... For the H.265 / HEVC standard, which has high decoding complexity, set... The value is between 0.7 and 0.9 to reduce the baseline budget and prevent decoding stuttering caused by high-complexity bitstreams.

[0058] Resource orchestration module 140 will score terminal capabilities. As input variables, the instantaneous budget value is calculated using a pre-defined nonlinear mapping function. This mapping mechanism is used to accelerate budget contraction at a nonlinear rate when terminal capabilities decline, thereby reserving a safety margin for computing power. Define the instantaneous budget value. as follows:

[0059] ;

[0060] in, For the current moment Terminal capability score; This is the computing power sensitivity index, and its value is greater than 1; Budget baseline values ​​are processed for macroblocks. When If a terminal's capability score drops even slightly, the instantaneous budget value will decrease more significantly. This configuration allows the system to immediately and drastically reduce the macroblock budget allocated to a terminal when it detects even slight signs of thermal throttling or increased decoding latency.

[0061] Because the terminal's operating status may fluctuate instantaneously due to factors such as background process scheduling, in order to avoid visually perceptible flickering or oscillations in video quality caused by sudden changes in computing power budget, the resource orchestration module 140 performs exponentially weighted moving average processing on the instantaneous budget value to generate the final effective macroblock processing budget. The calculation formula is:

[0062] ;

[0063] in: This represents the macroblock processing budget calculated at the previous time step; This is a smoothing factor, with a value ranging from 0 to 1; This is the instantaneous budget value. A smaller smoothing factor makes the budget change curve smoother, ensuring the continuity of video stream bitrate and resolution changes.

[0064] Resource orchestration module 140 will calculate the macroblock processing budget. Output to gradient differential encoding module 150. This is used as a rigid constraint threshold in the subsequent ROI mask generation process. When the gradient differential coding module 150 plans the high-quality and medium-quality regions of a single frame image, it needs to calculate the cumulative macroblock decoding overhead consumed in these regions and ensure that the total overhead does not exceed the macroblock processing budget. If it exceeds this rigid constraint threshold, the system will forcibly implement a region cropping or degradation strategy, thus realizing adaptive flow control based on edge capabilities.

[0065] The interaction normalization module 230 establishes a unified normalized logical coordinate system to shield the physical differences between heterogeneous client terminals 200 in terms of screen resolution, pixel density, and input device type. This module converts different types of input events into standard geometric coordinates relative to the video rendering area, ensuring that the server system 100 can generate ROI masks based on a unified spatial reference.

[0066] Interactive normalization module 230 defines a two-dimensional unit Cartesian coordinate system. The coordinate system's value ranges from [0, 1] on both the horizontal and vertical axes. The origin (0, 0) of this coordinate system is mapped to the top-left vertex of the video rendering container, and the coordinate (1, 1) is mapped to the bottom-right vertex of the video rendering container. All positional information is projected onto this unit space.

[0067] When the heterogeneous client terminal 200 is a device equipped with a mouse, touchscreen, or drawing tablet, the interaction normalization module 230 listens for pointer events dispatched by the browser kernel. The interaction normalization module 230 reads the clientX and clientY properties from the event object, which are used as the horizontal absolute pixel coordinates when the input event is triggered. Vertical absolute pixel coordinates when the input event is triggered Simultaneously, this module calls the `getBoundingClientRect()` method of the video rendering container DOM element to obtain the container's layout properties relative to the current viewport, including the normalized coordinates of the current focus. The calculation is as follows:

[0068] ;

[0069] in: The absolute pixel coordinates of the input event when it is triggered; The vertical absolute pixel coordinates when the input event is triggered; The horizontal pixel offset of the video rendering container relative to the top-left corner of the viewport; This is the vertical pixel offset of the video rendering container relative to the top-left corner of the viewport, corresponding to the getBoundingClientRect().left and top properties; , These are the rendering width and height of the video rendering container on the current device, corresponding to the getBoundingClientRect().width and height properties.

[0070] When the heterogeneous client terminal 200 is a device equipped with an infrared remote control, game controller, or directional keys, the interaction normalization module 230 adopts a focus tracking mechanism based on the Document Object Model (DOM). When the user switches focus using the directional keys, the browser kernel triggers a focus change event. The interaction normalization module 230 captures the DOM element currently receiving focus and also calls the element's getBoundingClientRect() method to obtain the geometric parameters of its bounding rectangle.

[0071] In this mode, the interaction normalization module 230 calculates the geometric center point of the current focus element and converts it into normalized coordinates. The calculation formula is:

[0072] ;

[0073] in: , These are the horizontal and vertical coordinates of the currently focused DOM element relative to the top-left corner of the viewport, respectively. , These are the pixel width and height of the currently focused DOM element, respectively; The horizontal pixel offset of the video rendering container relative to the top-left corner of the viewport; The vertical pixel offset of the video rendering container relative to the top-left corner of the viewport; , These represent the rendering width and height of the video rendering container on the current device. Discrete UI element selection operations are translated into specific geometric coordinates on the video stream plane.

[0074] Interactive normalization module 230 calculates Boundary truncation is performed to ensure that the final output x and y coordinate values ​​are strictly limited to the [0,1] interval. Corrected normalized coordinates. The payload segment written into the signaling data packet is sent to the server system 100 as the current exact focus.

[0075] When the heterogeneous client terminal 200 is in discrete input mode (such as using a remote control, keyboard arrow keys or game controller), the interaction normalization module 230 identifies interactive nodes around the current focus by executing topology prediction logic and constructs an enhanced coordinate dataset containing the prediction intent.

[0076] The interaction normalization module 230 traverses the Document Object Model (DOM) tree of the current page and selects a set of elements with the focusable attribute. The filtering criteria include: the element tag belongs to a preset set of interactive tags, or the element explicitly defines a tabindex attribute with a non-negative value. Simultaneously, the interaction normalization module 230 checks the element's calculated style, excluding unusable elements with a display attribute of none, a visibility attribute of hidden, an opacity attribute of 0, or HTML attributes containing disabled. To further eliminate visually obscured elements, the interaction normalization module 230 obtains the screen geometric center coordinates of each candidate element and calls document.elementFromPoint() to determine if the top-level element at that coordinate is the current candidate element itself or its child node; if not, the element is determined to be obscured and removed from the element set. Remove from the set of elements. elements in The module calls the getBoundingClientRect() method to obtain its geometric information in the viewport.

[0077] Interaction normalization module 230 is based on the currently focused element The position is used to calculate the element set in each of the four main navigation directions (up, down, left, and right). medium elements Navigation costs. Simulate the browser kernel's spatial navigation algorithm to determine the next target most likely to be selected when the user presses the arrow keys. Navigation direction vector. Navigation rating function The navigation scoring function A lower score indicates a higher probability that the element will be selected. The calculation formula is:

[0078] ;

[0079] in: This represents the navigation direction vector, with values ​​ranging from (0, ...). 1), (0,1), ( (1,0) or (1,0); express Compared to Perpendicular to the navigation direction The orthogonal distance on. Specifically, this distance is... Geometric center point and elements The geometric center point is perpendicular to The absolute value of the coordinate difference on the coordinate axes;

[0080] express Compared to Parallel to the navigation direction The projected distance on the surface. Specifically, this distance is... Geometric center point and elements The geometric center point is parallel to The absolute value of the coordinate difference on the coordinate axes. In particular, if Not here Along direction Within the defined half-plane, then The value can be positive infinity;

[0081] This is the orthogonal distance weighting coefficient, used to control the impact of orthogonal deviation on navigation scoring; This is the projection distance weighting coefficient, used to control the impact of projection distance on navigation scoring. It is typically set as follows: This is to reflect that users tend to select geometrically aligned elements when using the arrow keys for navigation.

[0082] The interaction normalization module 230 traverses the four directions and selects one or more elements with the smallest navigation scoring function value as potential focus elements. The interaction normalization module 230 extracts the geometric center points of these potential focus elements and calculates their corresponding normalized coordinates.

[0083] System defines topology prediction vectors The topology prediction vector It contains a weighted combination of the current exact focus and the predicted potential focus. Its data structure is represented as follows:

[0084] ;

[0085] in, Normalized coordinates for the current exact focus; , , , These are the normalized coordinates of the predicted nearest neighbor elements in each of the four directions. If there are no valid candidate elements in a certain direction, the corresponding coordinates are empty. Weighted by the level of attention currently focused on; To predict the attention weight of neighboring nodes.

[0086] The interactive normalization module 230 serializes the topology prediction vector and sends it to the server system 100 via the signaling interaction channel 320. Upon receiving this data, the server system 100 generates a non-uniform quantization parameter matrix in the gradient differential coding module 150. Specifically, the server uses... Generate a high-quality core ROI region centered on the image, and with , A pre-loaded area of ​​secondary image quality is generated centered on the predicted point. This mechanism ensures that when the user actually presses the directional keys on the remote control to switch focus, the video image of the target area is already sharp, thus avoiding visual blur.

[0087] When the heterogeneous client terminal 200 is in continuous input mode (such as mouse movement on a PC, touchpad sliding, or touchscreen dragging on a mobile device), the interaction normalization module 230 uses a kinematic trajectory extrapolation algorithm to predict the future interaction focus position based on historical sampling points in order to compensate for the system latency caused by network signaling transmission and video encoding.

[0088] Because the browser triggers mousemove or touchmove events extremely frequently, directly transmitting all coordinates would increase the load on the signaling channel. The interaction normalization module 230 maintains a circular queue in memory. Whenever a new input event is detected, the module extracts the normalized coordinates. and timestamp Store in a queue. If the queue is full, remove the oldest sampling point.

[0089] Interactive normalization module 230 based on circular queue The instantaneous velocity vector of the current cursor movement is calculated using the two or more most recent sampling points. To reduce calculation errors caused by minor jitter in the input device, a weighted average method is used to calculate the smoothed velocity vector. The calculation formula is as follows:

[0090] ;

[0091] in, Normalized coordinates; These are the normalized coordinates from the last sampling time; The timestamp of the current sampling moment, in milliseconds; The timestamp of the last sampling time; This is the smoothed velocity vector calculated at the previous moment; This is the speed update coefficient, with a value ranging from 0 to 1. A larger coefficient makes the speed calculation more sensitive to the current action, while a smaller coefficient provides better anti-jitter performance.

[0092] The interaction normalization module 230 predicts the target location that the user's focus might reach after system delay, based on the current normalized coordinates and smoothed velocity vector. The predicted target coordinates are defined. as follows:

[0093] ;

[0094] in, For forward-looking time windows; Normalized coordinates; To smooth the velocity vector, the interaction normalization module 230 periodically sends heartbeat packets through the signaling interaction channel 320 and parses the round-trip network latency (RTT) and the server's current average encoding time from the server's response data, using the sum of the two as the velocity vector. The value to be set. For example, if the current RTT is 50ms and the average encoding time is 15ms, then set to... =65ms. When the server... When the generated video frames are sent back to the client, the high-quality areas in the video frame can coincide with the position of the user's cursor after it has been moved.

[0095] Because linear extrapolation may cause predicted coordinates to exceed video boundaries, the interactive normalization module 230 pairs... The horizontal and vertical coordinate components are truncated and restricted to the [0,1] interval. Furthermore, to avoid frequent ROL redraws when the cursor is stationary or making slight movements, the interactive normalization module 230 calculates the magnitude of the smooth velocity vector. Only if the magnitude of the smooth velocity vector is... Only when the motion static zone threshold is exceeded will... Encapsulate and send as the focus of the predicted intent; otherwise, only send. As a static focus, the system effectively compensates for operational latency in cloud-rendered scenarios through the above methods.

[0096] The gradient differential coding module 150, based on real-time focus coordinates and predicted intent data received from the client, spatially divides each frame of the video image to be encoded into three regions with different coding quality weights. This partitioning mechanism aims to concentrate limited transmission bandwidth and decoding computing power on the regions most sensitive to the user's vision.

[0097] Gradient differential coding module 150 parses normalized coordinates in signaling data packets Using this normalized coordinate system Using the coordinates of the point as the geometric center, a rectangular region is generated as the core high-definition region. This core high-definition area Corresponding to the imaging range of the user's fovea, the requirement is to maintain the original image quality or use the lowest possible level of compression. Core high-quality area. The mathematical definition in the normalized coordinate system is as follows:

[0098] ;

[0099] in, The normalized coordinates of any pixel in the video frame; , These represent the normalized position values ​​of any pixel in the horizontal and vertical directions, respectively. , For the current normalized coordinates The horizontal and vertical components; , These are the normalized width and height thresholds for the core high-definition region, respectively. This represents absolute value operations. These two thresholds are dynamically set by the server based on the negotiated total video resolution and aspect ratio to ensure that the physical pixel area they cover conforms to the preset human eye focusing range.

[0100] Gradient differential coding module 150 extracts topology prediction vectors The set of coordinates of all potential focal elements contained therein (if in continuous input mode, extract the coordinates of the predicted target). Predicting the transition region. It consists of the union of extended regions generated around these prediction points, and subtracts the values ​​of the points from it. Overlapping areas. This region corresponds to the user's near peripheral vision or the target of an upcoming eye jerk, requiring moderate image clarity to ensure clear visibility when the gaze quickly switches to this area. Predicted transition areas. The defining formula is:

[0101] ;

[0102] in, The number of valid prediction points; For the first Prediction points A rectangular bounding box centered on the region; defining the region expansion factor as... ,but The width is set to The height is set to ; For the core high-definition area; Represents the iterative form of the set union operation; This represents the set difference operation, ensuring that regions do not contain each other. The system pre-loads surrounding image details that the user is highly likely to click or move to.

[0103] Except for video frames and The remaining portion other than the specified area is defined as the background suppression region. Background suppression region Corresponding to the user's far peripheral visual range, the human eye is not sensitive to high-frequency details in this area. The gradient differential coding module 150 will implement a high-intensity compression strategy for this area.

[0104] ;

[0105] in, The background suppression region; Frame represents the complete video frame planar region; This represents the set difference operation; For the core high-definition area; To predict the transition region; This indicates the set union operation.

[0106] The gradient differential coding module 150 assigns basic quantization parameters (QP) to the three regions mentioned above. A smaller QP value results in higher image quality and a higher bit rate. The system sets three baseline QP values: , and ,satisfy .

[0107] To adapt to the block structure characteristics of video coding standards, the gradient differential coding module 150 first obtains the macroblock size configured by the current encoder. The video frame is then divided into a macroblock grid. For the first... macroblock According to macroblocks The region where the geometric center is located determines the macroblock. Initial QP value :

[0108] ;

[0109] in, Indicates the first Normalized center coordinates of each macroblock; Represents the first video frame after the video frame is divided. One macroblock; Indicates the first The coordinates of the geometric center point of each macroblock; Indicates belonging to; , , These represent the preset quantization parameter values ​​for the core high-definition region, the predicted transition region, and the background suppression region, respectively. For the core high-definition area; To predict the transition region; This represents the background suppression region. This mapping process generates a QP Map (Quantization Map Table) aligned with the video macroblock grid. This table is directly input to the bitrate control unit of the video encoder, guiding the encoder to perform differentiated quantization operations on the transform coefficients of different macroblocks, thereby achieving non-uniform compression based on visual attention.

[0110] While topology prediction vectors can provide multiple potential interaction focal points, in scenarios with limited computing power on heterogeneous clients or fluctuating network bandwidth, simultaneously rendering all prediction regions at high quality may lead to increased encoding latency or decreased frame rate. To address this, the resource orchestration module introduces a dynamic filtering mechanism based on computing power budget, which prunes or merges regions in the prediction list while ensuring basic display requirements are met.

[0111] The resource orchestration module obtains real-time status parameters of the server-side video encoder by reading its status register or listening to the encoding completion callback interface. These parameters include the current encoding queue length, the actual encoding time of the previous frame, and the CPU / GPU load rate. Based on these parameters, it calculates the available computing power budget factor for the current frame. This computing power budget factor It is a normalized value that reflects the system's ability to process additional high-quality macroblocks. Its calculation formula is as follows:

[0112] ;

[0113] in, This is the computing power budget factor; This represents the function that takes the maximum value. This represents the total capacity of a unit time window; This represents the actual encoding time of the previous video frame. This represents the queuing time in the current encoding queue. The frame interval time corresponding to the target frame rate; Indicates the weighted load factor; This is a system safety redundancy factor, used to reserve computing power to cope with sudden changes in screen complexity. If A value close to 0 indicates that the system load is saturated and no longer suitable for handling additional prediction areas.

[0114] The resource orchestration module traverses each prediction point in the topology prediction vector. Calculate the prediction point The centered rectangular bounding box The number of macroblocks covered will be defined as the encoding cost. .

[0115] At the same time, combined with the transition probabilities recorded in the topology network Calculate the first Preservation priority score for each predicted region :

[0116] ;

[0117] in, Indicates the transition probability; For encoding cost; This represents the square root operation of the encoding cost. This formula reflects the selection principle of prioritizing high-probability, low-cost methods. According to... Sort all predicted regions from highest to lowest.

[0118] The resource orchestration module determines the maximum number of prediction regions or the total area limit allowed to be retained in this frame based on the calculated computing power budget factor. The module selects the first, sorted regions in sequence. There are candidate regions such that the total encoding cost of the selected region satisfies the following constraint:

[0119] ;

[0120] in, This indicates a summation operation; The number of valid prediction points after pruning (i.e. ); For the sorted number The encoding cost of each candidate region; Indicates the computing power budget factor; This is the maximum theoretical macroblock cost constant, which is preset according to the server hardware specifications. (Failed to enter the preceding section) The predicted points will be discarded, and their corresponding screen areas will subsequently be assigned to the background suppression region. deal with.

[0121] In determining the retention After predicting each region, in order to reduce the additional overhead caused by defining multiple independent slices or tile header information in the video stream, the resource orchestration module checks the spatial proximity between each region.

[0122] For any two preserved prediction regions and First calculate and minimum bounding rectangle Assuming The coordinates of the top left corner are The coordinates of the lower right corner are , The corresponding coordinates are and ,but top left corner coordinates and the coordinates of the bottom right corner The calculation is as follows:

[0123] ;

[0124] in, This indicates the minimum value operation; This indicates the operation of finding the maximum value.

[0125] Then, the area of ​​the circumscribed rectangle is calculated. If the following region merging threshold condition is met, the two regions are merged into a single rectangular region:

[0126] ;

[0127] in, This indicates the calculation of the pixel area of ​​a rectangular region; This is the threshold for region fusion; Indicates two candidate regions and The minimum bounding rectangle. This formula states that if the increase in blank area relative to the total area of ​​the original region after merging does not exceed a preset threshold, then merging is performed. Logically, the merging operation will... and Replace with The updated region list is then output to the gradient difference coding module as the generated region list. The system uses basic data. Through a fusion mechanism, the system reduces the complexity of the bitstream structure and the overhead of the signaling header while slightly increasing the number of macroblocks to be encoded.

[0128] After determining the spatial region affiliation of each macroblock, the gradient differential coding module converts these geometric attributes into low-level control parameters recognizable by the video coding standard. This embodiment generates the final encoded quantization parameter map by constructing a multi-dimensional quantization parameter offset model, combining motion and texture characteristics.

[0129] Based on the generated region segmentation results, the system assigns each macroblock in the current frame a region. Assign a basic quantization parameter It should be noted that although this embodiment uses a 16×16 macroblock of the H.264 standard as an example, under the H.265 / HEVC standard, this macroblock corresponds to a 64×64 coding tree unit or a recursively divided coding unit. The basic quantization parameters are directly mapped to the region type where the center point of the macroblock is located. , or ), that is, the initial quantization parameter, whose value is set according to The stepped distribution establishes the spatial hierarchy of image quality.

[0130] Due to the temporal masking effect of the human visual system, the human eye's ability to perceive details of high-speed moving objects decreases. The module reads motion vectors generated by the video encoder or calculated using optical flow. For the... macroblock Calculate the motion adaptive QP offset The calculation formula is as follows:

[0131] ;

[0132] in, For the first macroblock The average motion vector includes a horizontal component. and vertical components ; The magnitude of the motion vector is calculated as follows: , indicating the magnitude of the speed of motion; This represents the natural logarithm operation, used for nonlinear compression of velocity values; This is the speed sensitivity coefficient, used to adjust the weight of the impact of motion speed on image quality degradation; This indicates a floor operation, ensuring the output is an integer QP step size. When the image block is stationary or slightly moving, the offset is close to 0; when the image block moves at high speed, this formula calculates a positive offset value.

[0133] Based on the human eye's quantization noise masking effect on complex textured regions and its sensitivity to noise in flat regions, the module analyzes the spatial frequency domain features of the image. It calculates the... macroblock Spatial texture complexity And calculate texture adaptive QP offset. The formula is as follows:

[0134] ;

[0135] in, For the current macroblock Spatial texture complexity; The average texture complexity of the current frame or current region; This is the texture adjustment coefficient; The minimum cutoff threshold; The maximum truncation threshold; This is a numerical truncation function used to restrict the calculation results to a specific range. Within the specified interval. For regions with complex textures, the calculation result is positive; for flat regions, the calculation result is negative.

[0136] By superimposing the calculation results of the base layer, motion layer, and texture layer, we obtain the... macroblock The target quantization parameters are finally transmitted to the hardware encoder. :

[0137] ;

[0138] in, Basic quantization parameters; and These are the motion-adaptive QP offset and the texture-adaptive QP offset, respectively. and This refers to the range of QP values ​​allowed by the video coding standard. This is a range clamping function that ensures the final parameters are within a valid range. Through this mapping logic, the system translates focus predictions into specific encoding instructions. The aforementioned parameter graph is injected into the encoding pipeline via the encoder API.

[0139] After generating the macroblock-level target quantization parameter map of the entire frame through the aforementioned modules, the encoding engine converts these parameters into bitstream syntax elements that conform to video compression standards, and performs post-processing on the reconstructed image to eliminate block artifacts caused by changes in quantization parameters.

[0140] The video coding standard transmits the difference between the QP value of the current macroblock and the predicted QP value. The system traverses each macroblock of the current frame in raster scan order or zigzag scan order. Calculate the quantization parameter differential syntax elements to be written to the bitstream. The calculation formula is as follows:

[0141] ;

[0142] in, Represents the first video frame after the video frame is divided. One macroblock; Indicates the target quantization parameter; Indicates the predicted quantization parameters; This represents the quantization parameter difference syntax element. It is a signed integer whose value is subsequently written into the video bitstream using Golomb coding or arithmetic coding. This ensures the generated bitstream conforms to the syntax specifications of a standard decoder, which reconstructs the target quantization parameter set at the encoder by superimposing the predicted and difference values.

[0143] Differences in quantization parameters exist between the region of interest and the background region, potentially causing block artifacts at the region boundaries. The system dynamically adjusts the control parameters of the block filter for each macroblock. Calculate the offset of its loop filter parameters. The calculation formula is as follows:

[0144] ;

[0145] in, This indicates a summation operation; Macroblock A set of spatial neighborhood macroblocks; This indicates the number of neighboring macroblocks involved in the calculation, used for normalization; Indicates the neighborhood macroblock in the set Index in; Representing neighboring macroblocks The target quantization parameters; This represents the absolute value operation; This represents the difference in quantization parameters between the current macroblock and its neighboring macroblocks; This is the filter strength gain coefficient, used to map the QP difference to a standard-defined filter offset; This indicates the rounding operation.

[0146] When a macroblock is within a region where QP changes gently, the offset is close to 0; when a macroblock is at the boundary between high and low image quality, the offset increases, indicating that the encoder applies a stronger smoothing filter.

[0147] After transforming, quantizing, and entropy encoding the residual data, the encoder encapsulates this data into Network Abstraction Layer Units (NAL Units). For hardware encoder interfaces that support region-of-interest (ROI) encoding, this is mapped to parameters in the configuration structure. Specifically, the calculated target quantization parameter matrix is ​​filled into the buffer specified by the hardware interface, and the hardware encoding core performs low-level differential calculations and filtering operations. At this point, the output bitstream contains non-uniform image quality distribution information based on gaze point prediction.

[0148] After the decoder outputs the reconstructed video frame, due to the non-uniform quantization strategy used at the encoding end, if the user's actual gaze point deviates from the gaze point predicted by the encoding end, or if the user observes a low-quality background area, the client rendering engine will compensate for the reconstructed image through a post-processing workflow.

[0149] The client device receives video frame data and accompanying metadata (including the predicted gaze coordinates used by the encoder). Meanwhile, the client obtains the user's actual gaze coordinates at the current moment through local eye-tracking sensors. The system calculates the Euclidean distance between two points. The calculation formula is as follows:

[0150] ;

[0151] in, These are the actual coordinates of the gaze point; To predict the coordinates of the gaze point; Euclidean distance; , These are the horizontal and vertical pixel coordinates of the actual gaze point, respectively. , These are the horizontal and vertical pixel coordinates for predicting the gaze point coordinates, respectively. This represents the Euclidean distance calculation. If the gaze deviation distance is less than the preset tolerance threshold, it indicates that the prediction is accurate, and rendering proceeds directly; if the gaze deviation distance is greater than the threshold, it indicates that the user is observing an area with low encoding quality, triggering a subsequent visual compensation mechanism.

[0152] The rendering engine uses the actual gaze coordinates. A dynamic desharpening mask is applied to the surrounding area. This applies to each pixel on the screen. Calculate the compensated pixel values The calculation formula is as follows:

[0153] ;

[0154] in, , These are the horizontal and vertical pixel coordinates of the currently processed pixel in the screen coordinate system. The original reconstructed pixel brightness value output by the decoder; These are the pixel values ​​after Gaussian blurring of the original reconstructed image; Represents the high-frequency detail components of an image; For the current pixel Coordinates of the actual gaze point The distance between them; The distance-dependent sharpening gain function is specifically defined as follows: ; This represents the maximum sharpening intensity, used to control the peak amplitude of sharpening. For the natural constant An exponential function with base 0; This is a parameter controlling the sharpening range, used to control the rate at which the sharpening intensity decreases with distance. A stronger sharpening enhancement is applied to the area the user is currently looking at to improve edge contrast; as distance increases, the sharpening intensity decreases Gaussian to avoid introducing noise in surrounding visual areas.

[0155] When the gaze deviation is large, the region boundary may fall into the user's central field of vision. To eliminate jumps between different quantization levels, the system performs a boundary fusion operation. For regions located on either side of the boundary line... Calculate the merged pixel value for each pixel within the width. The formula is as follows:

[0156] ;

[0157] in, These are pixel values ​​reconstructed using lower quantization parameters; These are pixel values ​​reconstructed using higher quantization parameters; The calculation method for the fusion weighting coefficient is as follows: ; For the current pixel The signed distance to the boundary line of the region (positive for regions with low quantization parameters); This is the preset width of the smooth transition band. Smooth transitions between regions are achieved through linear interpolation.

[0158] Considering the latency from image acquisition, encoding, transmission to decoding, the client reprojects the image using the latest head pose information before rendering it to the screen. The system constructs a mesh model, maps the processed image texture onto this mesh, and transforms the mesh vertex coordinates based on the difference between the current head quaternions and the head quaternions at the time of frame rendering. This ensures that the user's perceived image remains spatially stable even with errors in gaze point prediction or network jitter.

[0159] The following is a specific implementation example of this solution:

[0160] Suppose a user uses a heterogeneous client terminal 200 (model A, for example, a three-year-old Android phone) to access the digital financial advisor function within a bank's app. In this case, the target video stream specification negotiated for this terminal is a resolution of 1920×1080 (i.e., width...). ,high ), target frame rate fps.

[0161] During the session, the client's performance telemetry module 220 periodically (e.g., once per second) collects the hardware status. Assuming at the current moment... The performance telemetry module 220 read the following raw data: Due to multiple applications running in the background, the current heap memory usage is 600MB, while the maximum heap memory allowed for this process by the system is 1000MB; the average video decoding latency over the past 30 frames was calculated. The timeframe increased to 40ms; meanwhile, the thermal state enumeration value returned by the underlying operating system corresponds to mild overheating, which is mapped to a thermal throttling state indicator factor. The performance telemetry module 220 immediately adjusts the settings according to the preset weights (set as follows). , , ) and reference decoding delay time ms computing terminal capability score The specific calculation process is as follows: First, calculate the memory component 1 - (600 / 1000) = 0.4; then calculate the latency component. Substituting into the formula, we get The score of 0.693 was encapsulated in a signaling packet and sent to the server.

[0162] After receiving the score, the resource orchestration module 140 of the server system 100 first calculates the macroblock processing budget baseline value. .for The video, using 16×16 pixels as macroblock units, has 120 macroblocks in the width direction and 67.5 macroblocks in the height direction (rounded up to 68), for a total of [number missing] macroblocks per frame. One. Baseline budget per second. A macroblock (set) Subsequently, the module, based on the received score of 0.693 and the preset computing power sensitivity index, Calculate instantaneous budget value The calculation process is as follows: One macroblock. This result indicates that due to the degradation of terminal performance, the server will force a limit on the number of macroblocks processed per second to within approximately 48% of the baseline value, which means that a large amount of regional degradation processing of the video image must be performed. Meanwhile, the client's interaction normalization module 230 detects that the user's finger is sliding on the screen, preparing to click the details button located in the lower right corner of the screen. The normalization module measures the current touch point coordinates. And utilize the smoothed velocity vector of the most recent 5 sampling points Predict the landing point coordinates after 60ms. These two coordinate data points are synchronized to the server. The server's gradient differential encoding module 150 uses this data to generate the QP Map (Quantization Parameter Mapping Table) for the current frame. The system first defines two core high-quality regions (RI): one is a region centered at the digital face center (0.5, 0.3) with a radius of 0.1 (normalized unit); the other is a region centered at the user-predicted landing point... The region centered at [center] has a radius of 0.1. Macroblocks within these two regions are assigned the lowest possible quantization parameter. The annular region immediately surrounding these areas is defined as the predicted transition region (RII), and is assigned... The remaining majority of the screen area (such as a static background wall) is designated as the background suppression region RIII, assigned the highest quantization parameter. .

[0163] In the specific encoding execution phase, assuming the digital human's hand waves rapidly while speaking, the macroblock containing that hand... Although located in the RII region (basic QP is 32), the encoder calculates its motion vector magnitude. Pixels. According to the motion adaptive formula (assuming...) ), calculate QP offset Therefore, the final target QP value for this fast-motion macroblock is 32 + 4 = 36. This dynamic adjustment utilizes the human eye's insensitivity to motion blur to further reduce the bitrate, while ensuring that the total macroblock overhead is strictly controlled within the calculated budget limit of 117,504 macroblocks per second. Finally, after the compressed video stream is transmitted back to the smart terminal A and decoded, the media rendering container module 240 performs visual compensation. Assume that the user's actual gaze point at this time... It falls exactly at the digital human's facial pixel (960, 324). For a pixel near this gaze point... Its decoded brightness value Background brightness after Gaussian blur (That is, a high-frequency detail value of 10). The system calculates the distance from this pixel to the gaze point. And calculate the gain coefficient based on the sharpening gain function. The final displayed pixel values ​​are compensated to Through this edge-side post-processing, even if the server sends compressed video frames due to computing power budget limitations, users can still perceive sharpened and enhanced clear image quality in the core areas they are interested in, thus achieving a smooth and clear digital human interaction experience on low-performance devices.

[0164] See attached document Figure 3 This graph visually illustrates the session state change curve of the smart terminal A (a three-year-old Android phone) mentioned in the embodiment during a 60-second session while running the digital human financial advisor function. The horizontal axis represents the session duration, and the left vertical axis represents the normalized terminal capability score. (Solid line), the right vertical axis represents the value relative to the reference value. The computing power budget percentage (dashed line). As the session progresses, due to multiple background applications running as described in the example, the heap memory usage gradually increases from 200MB to 600MB, and video decoding latency... From 20ms to 40ms, the terminal score, shown by the solid line, exhibits a continuous downward trend. Particularly around the 25th second, the device triggers thermal throttling and frequency reduction as mentioned in the embodiment; the thermal throttling status indicator factor... The score plummeted from 1.0 to 0.8, causing a drop in the terminal capability rating. A significant step-like decline occurred. The 40th second, marked by the vertical dashed line in the figure, is the current time described in the example. At this point, the data collected by the performance telemetry module is used to calculate the terminal capability score. (Corresponding to the square markers on the solid lines in the diagram). The server-side resource orchestration module is based on a preset computing power sensitivity index. A non-linear mapping was applied to the score, causing the computing budget (represented by the dashed line) to decrease at a steeper rate than the score itself. Ultimately, at the 40-second mark, the computing budget percentage was limited to approximately 48.0% (corresponding to the circular marker on the dashed line in the graph). This state visually demonstrates the system's defensive reduction strategy: when it detects terminal memory pressure, increased latency, or hot throttling, it proactively reduces the coding load to less than half of the baseline value to prevent application crashes or stuttering.

Claims

1. A multi-client-based digital human display and interaction system, characterized in that, It includes a server system (100), a communication network (300), and heterogeneous client terminals (200). The heterogeneous client terminal (200) is used to collect runtime hardware status data and generate terminal capability scores, as well as capture user input operations and convert them into virtual focus data, and send the virtual focus data to the server system (100) through the communication network (300). The server system (100) includes a resource orchestration module (140) and a gradient differential coding module (150). The resource orchestration module (140) is used to receive the terminal capability score and calculate the computing power budget for the current connection based on the terminal capability score. The gradient differential coding module (150) is used to receive the original video frame, divide the original video frame into regions with different image quality weights according to the virtual focus data, and generate a compressed video stream with non-uniform quality distribution in combination with the computing power budget. The heterogeneous client terminal (200) is also used to receive and decode the compressed video stream and render the decoded image frames to the display device.

2. The digital human display and interaction system based on multiple clients according to claim 1, characterized in that, The heterogeneous client terminal (200) includes a performance telemetry module (220): The runtime hardware status data periodically collected by the performance telemetry module (220) includes at least heap memory occupancy, video decoding latency, and device thermal throttling status. The performance telemetry module (220) uses a weighted normalization model to calculate the heap memory occupancy rate, the video decoding delay time, and the device hot throttling state to obtain the terminal capability score. When the video decoding delay time exceeds the target frame interval time, the terminal capability score will decrease.

3. The digital human display and interaction system based on multiple clients according to claim 1, characterized in that, The heterogeneous client terminal (200) includes an interaction normalization module (230): The interactive normalization module (230) is used to establish a unit Cartesian coordinate system, project the signal from the input device in discrete input mode or continuous input mode into the unit Cartesian coordinate system, and generate normalized coordinates as the virtual focus data; When the input device is in the discrete input mode, the interaction normalization module (230) traverses the document object model tree of the current page, filters the focusable elements, calculates the navigation score based on the navigation direction, and predicts the potential interaction area.

4. The digital human display and interaction system based on multiple clients according to claim 3, characterized in that, When the input device is in the continuous input mode, the interaction normalization module (230) adopts a kinematic trajectory extrapolation algorithm; The interactive normalization module (230) maintains a circular queue of historical sampling points, calculates a smooth velocity vector, and determines the look-ahead time window by combining the round-trip network latency and the average encoding time of the server. The interactive normalization module (230) uses the smoothed velocity vector and the look-ahead time window to calculate the predicted target coordinates, and uses the predicted target coordinates as the virtual focus data containing the prediction intention.

5. A digital human display and interaction system based on multiple clients according to claim 1, characterized in that, The process by which the resource orchestration module (140) calculates the computing power budget includes: Using a nonlinear mapping function, the terminal capability score is mapped to an instantaneous budget value, and the decay rate of the instantaneous budget value as the terminal capability score decreases is controlled by a computing power sensitivity index. The instantaneous budget value is processed by an exponentially weighted moving average to generate the final effective computing power budget.

6. A digital human display and interaction system based on multiple clients according to claim 1, characterized in that, The gradient differential coding module (150) divides the original video frame into a core high-quality region, a prediction transition region, and a background suppression region in space based on the virtual focus data. The core high-definition area is a rectangular area with the coordinates of the virtual focus data as its geometric center. The predicted transition region is the region formed by subtracting the core high-quality region from the extended region generated by the prediction points. The background suppression region is the remaining part of the original video frame excluding the core high-quality region and the predicted transition region; The gradient differential coding module (150) assigns incremental basic quantization parameters to the core high-quality region, the predicted transition region, and the background suppression region.

7. A digital human display and interaction system based on multiple clients according to claim 6, characterized in that, The gradient differential coding module (150) is also used to generate a macroblock-level target quantization parameter map: The gradient differential coding module (150) determines the region type to which the macroblock belongs and obtains the initial quantization parameters; Calculate the motion vector magnitude of the macroblock and generate motion adaptive quantization parameter offsets based on the motion speed; Calculate the spatial texture complexity of the macroblock and generate texture adaptive quantization parameter offsets based on texture features; The initial quantization parameters, the motion-adaptive quantization parameter offset, and the texture-adaptive quantization parameter offset are superimposed and clamped to obtain a macroblock-level target quantization parameter map.

8. A digital human display and interaction system based on multiple clients according to claim 6, characterized in that, The resource orchestration module (140) also executes a dynamic coding constraint strategy based on the computing power budget: Calculate the available computing power budget factor for the current frame, which is related to the coding queue length and the coding time of the previous frame; Calculate the retention priority score based on the coding cost and transition probability of the predicted region; The predicted regions are sorted and filtered according to the retention priority score to ensure that the total encoding cost of the retained predicted regions does not exceed the threshold limited by the computing power budget factor; For the retained predicted regions, if the area ratio of the merged blank regions does not exceed a preset threshold, then a region fusion operation is performed.

9. A digital human display and interaction system based on multiple clients according to claim 1, characterized in that, The heterogeneous client terminal (200) also includes a media rendering container module (240). The media rendering container module (240) performs visual compensation during rendering, calculating the deviation distance between the actual gaze point and the predicted gaze point at the encoding end; If the deviation distance exceeds the threshold, a dynamic desharpening mask is applied to the area around the actual gaze point, with the sharpening intensity decreasing Gaussian with distance.

10. A multi-client-based digital human display and interaction system according to claim 1, characterized in that, The server system (100) also includes a digital human rendering engine module (110), a streaming media service module (120), and a signaling control module (130). The signaling control module (130) transmits the terminal capability score and the virtual focus data to the heterogeneous client terminal (200) via a WebSocket or WebRTC data channel; The digital human rendering engine module (110) synthesizes the original audio and video data stream according to the driving instructions of the signaling control module (130) and inputs it to the gradient differential encoding module (150). The streaming media service module (120) is used to transmit the compressed video stream after it has been compressed by the gradient differential encoding module (150).

Images