Video conference interaction equipment with self-adaptive energy-saving control

Abstract

The invention relates to the technical field of video conferences and multimedia signal processing, and discloses video conference interaction equipment with self-adaptive energy-saving control, which comprises a central processing unit, an eye movement tracking sensor, a video processing unit and a display driving controller, the energy consumption arbitration controller module calculates a real-time comprehensive priority weight according to the semantic metadata and a fixation point coordinate, and for a video stream of which the real-time comprehensive priority weight is lower than a preset threshold value, the virtualized coding and decoding interface layer module intercepts a predictive coding frame and injects the predictive coding frame into a virtual null-hop unit constructed into a full-skip mode; triggering the video processing unit to execute zero copy address mapping operation, bypass inverse quantization and inverse transformation operation; the display driving controller synchronously reduces the refresh rate and the backlight current of the corresponding display area, and the power consumption of equipment is reduced while the decoding context continuity is maintained through cooperative control of virtualized null-hop injection and partition display.

Description

Technical Field

[0001] This invention relates to the field of video conferencing and multimedia signal processing technology, specifically to a video conferencing interactive device with adaptive energy-saving control. Background Technology

[0002] Modern video conferencing devices are widely used in remote work and collaboration scenarios. They need to receive and process multiple high-resolution video streams simultaneously to build an immersive meeting environment. To maintain the smoothness and clarity of the image, the central processing unit and video processing unit in the video conferencing device perform complete entropy decoding, inverse quantization, inverse transformation, and motion compensation operations on each network abstraction layer unit received. The reconstructed image data is then transmitted to the display driver controller. The display panel always maintains a globally consistent high refresh rate and high-brightness backlight drive current to ensure that the image content of all video channels is output and displayed at the highest quality standard.

[0003] However, the human visual system has a foveal imaging characteristic, which means that users can only maintain a high-definition visual perception of a local area on the display panel at a specific time, while the ability to perceive the surrounding area is reduced. Existing video conferencing interactive devices adopt a symmetrical processing architecture, which allocates completely equal computing and power resources to the focal area that the user is looking at and the non-focused background area. This undifferentiated full decoding and global high-specification display driving strategy results in a large amount of redundant computing and ineffective energy consumption in the non-focused area. The continuous high-load operation not only increases the overall power consumption and heat accumulation of the video conferencing interactive device, but also shortens the continuous working time of battery-powered devices.

[0004] By constructing an asymmetric video stream processing mechanism, deep decoding-level and display-level energy-saving control is implemented for video streams in non-interested areas without interrupting the video decoding context logic. This is of great value in improving the energy efficiency ratio of the device and optimizing the user experience. Therefore, this invention proposes a video conferencing interactive device with adaptive energy-saving control to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a video conferencing interactive device with adaptive energy-saving control, which solves the problems of excessive power consumption of the central processing unit and display panel caused by full decoding and full-screen high refresh rate display when processing multiple high-definition video streams, as well as the inability to dynamically adjust energy consumption allocation according to the user's visual focus.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: a video conferencing interactive device with adaptive energy-saving control, comprising a central processing unit, a network interface unit, an eye-tracking sensor, a video processing unit, a display driver controller, a display panel, a memory, a central processing unit operating semantic and gaze perception module, and an energy consumption arbitration controller module;

[0007] A virtualized codec interface layer module is provided between the data output end of the network interface unit and the data input end of the video processing unit;

[0008] The semantic and gaze perception module parses semantic metadata from the real-time transmission protocol data packets received from the network interface unit, and obtains gaze point coordinates and eye movement velocity vectors from the eye tracking sensor;

[0009] The energy consumption arbitration controller module calculates the real-time comprehensive priority weight for each video stream based on semantic metadata and gaze point coordinates, and compares the real-time comprehensive priority weight with a preset threshold. When the real-time comprehensive priority weight is greater than or equal to the preset threshold, the energy consumption arbitration controller module generates a first control signal. When the real-time comprehensive priority weight is less than the preset threshold, the energy consumption arbitration controller module generates a second control signal.

[0010] The virtualization codec interface layer module receives a first control signal or a second control signal. When the first control signal is received, the virtualization codec interface layer module transmits the corresponding video stream data to the video processing unit. When the second control signal is received, the virtualization codec interface layer module intercepts the predictive coding frames and bidirectional predictive coding frames in the corresponding video stream data and generates a virtual skip unit, which is then injected into the video processing unit. The virtual skip unit is a network abstraction layer unit that does not contain image residual data, and the virtual skip unit is constructed in a full skip mode.

[0011] Based on real-time comprehensive priority weights, the display driver controller reduces the refresh rate and backlight current of the display area corresponding to the second control signal.

[0012] Preferably, the energy consumption arbitration controller module executes the following weight calculation logic: obtains the semantic weight balance coefficient, and determines the semantic weight value based on the audio level indication information and the active speaker identification information; determines the gaze weight value based on the geometric positional relationship between the gaze point coordinates and the display area rectangle of the video stream on the display panel; performs a weighted summation calculation on the semantic weight value and the gaze weight value to obtain the real-time comprehensive priority weight, wherein, when the gaze point coordinates are located inside the display area rectangle, the gaze weight value takes the maximum normalized value; when the gaze point coordinates are located outside the display area rectangle, the gaze weight value is calculated by exponential decay based on the Euclidean distance from the gaze point coordinates to the display area rectangle.

[0013] Preferably, the energy consumption arbitration controller module also monitors the magnitude of the eye movement velocity vector; when the magnitude of the eye movement velocity vector exceeds the preset physiological saccade threshold, the energy consumption arbitration controller module determines that it has entered the saccade inhibition period, and calculates the predicted landing area based on the eye movement velocity vector; if the video stream corresponding to the predicted landing area is currently under the control of the second control signal, the energy consumption arbitration controller module generates a network preemption trigger signal, instructing the network interface unit to send a media stream feedback control message to the remote server, the media stream feedback control message being an image loss indication message or an intra-frame request message.

[0014] Preferably, the logic for the energy consumption arbitration controller module to calculate the predicted landing point area includes: obtaining the gaze coordinates, eye movement velocity vector, and eye movement acceleration vector at the current sampling time and determining the predicted time span of the saccade process; based on the gaze coordinates, eye movement velocity vector, and eye movement acceleration vector, using a kinematic extrapolation model to calculate the coordinates of the predicted landing point area in the pixel coordinate system of the display panel. The timing of the energy consumption arbitration controller module sending the media stream feedback control message is configured to satisfy the following timing constraints: the sum of the trigger time, network round-trip latency, remote encoding processing time, and local decoding processing time is less than or equal to the sum of the saccade action start time and the duration of the saccade suppression state.

[0015] Preferably, the virtualized codec interface layer module includes a gated routing submodule; the gated routing submodule performs header parsing on the input network abstraction layer unit to identify the sequence parameter set, image parameter set, instant decoding refresh frame and predictive coding frame; when the second control signal is received, the gated routing submodule directs the sequence parameter set, image parameter set and instant decoding refresh frame to the pass-through path to enter the video processing unit, and directs the predictive coding frame to the interception path for discarding, while triggering the generation logic of the virtual skip unit.

[0016] Preferably, the virtualized codec interface layer module further includes a virtual unit generation submodule. The virtual unit generation submodule constructs a virtual skip unit. The construction steps include: extracting the frame sequence number and image display order count of the intercepted original network abstraction layer unit, and writing the extracted frame sequence number and image display order count into the slice header of the virtual skip unit; constructing the syntax structure of the full skip mode, setting the macroblock skip operation count in the virtual skip unit to the total number of macroblocks in the current video frame, or setting the skip flag position of the coding tree unit to a valid state; the syntax structure of the full skip mode instructs the video processing unit to use zero motion vectors and a reference frame index pointing to the previous frame for decoding.

[0017] Preferably, the video processing unit responds to the virtual skip unit by performing a zero-copy address mapping operation; the zero-copy address mapping operation includes: directly pointing the output memory address pointer of the current decoded frame to the physical memory address of the reconstructed image of the previous frame, bypassing the inverse quantization and inverse transform operations of the texture reconstruction engine inside the video processing unit, and only updating the global state variables inside the decoder to maintain the continuity of the decoding context.

[0018] Preferably, the display driver controller integrates a coordinate mapping logic unit; the coordinate mapping logic unit stores the physical backlight partition layout parameters of the display panel, and calculates the overlap area between the display area of ​​the video stream and the backlight control area. When the overlap area exceeds a preset coverage ratio, a mapping relationship between the video stream and the backlight control area is established; according to the mapping relationship, the display driver controller outputs a reduced drive current for the backlight control area of ​​the video stream that is associated with the second control signal control state.

[0019] Preferably, the logic for reducing the output drive current of the display driver controller includes: obtaining the maximum rated drive current, the content adaptive brightness gain, and the brightness drive coefficient; calculating the product of the maximum rated drive current, the content adaptive brightness gain, and the brightness drive coefficient to obtain the drive current; wherein, when the video stream associated with the backlight control area is in a state controlled by the second control signal, the brightness drive coefficient is set to a preset low-power mode brightness attenuation factor.

[0020] Preferably, the display driver controller activates the vertical synchronization mask logic; for the video stream in the state controlled by the second control signal, the display driver controller intercepts the video memory read requests for the video stream layer according to the preset down-rate ratio, reads new data of the video stream only in the scan cycle that meets the down-rate ratio, and keeps the liquid crystal molecules of the display panel unchanged in the other scan cycles.

[0021] This invention provides a video conferencing interactive device with adaptive energy-saving control. It has the following beneficial effects:

[0022] 1. This invention sets up a virtualized codec interface layer module between the network interface unit and the video processing unit. This module can intercept predictive coding frames for video streams in non-focused regions and inject virtual skip units constructed in full skip mode. This allows the video processing unit to directly perform zero-copy address mapping operations and bypass the inverse quantization and inverse transform operations of the texture reconstruction engine when decoding low-priority video streams. This reduces the computational load and dynamic power consumption of the central processing unit and the video processing unit while maintaining the continuity of the decoder context and without destroying the syntactic integrity of the video stream.

[0023] 2. This invention utilizes an energy consumption arbitration controller module to monitor the eye movement velocity vector. It can calculate and predict the landing point area in advance during the physiological gap of rapid eye saccades and trigger a network preemption mechanism. By completing network request sending, remote encoding, and local decoding preparation in parallel within the short time window of the saccade action, it ensures that the video stream has been restored to a high-definition state the moment the human eye's gaze is stably placed on the new area of ​​interest. This solves the problem of image blurring or delay waiting caused by traditional low-power solutions when the gaze is switched, and achieves a seamless visual experience switch.

[0024] 3. This invention combines the coordinate mapping logic unit and vertical synchronization mask logic inside the display driver controller to achieve energy-saving control at the physical layer of the display panel. By performing current attenuation on the backlight partition of non-interested areas and refreshing rate interception on the display layer according to real-time comprehensive priority weight, the video conferencing interactive device not only saves energy on the data processing side, but also reduces power consumption on the photoelectric conversion and display driving side, extending the device's battery life in multi-channel concurrent video scenarios. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall hardware architecture of the system of the present invention;

[0026] Figure 2 This is a schematic diagram of the energy consumption arbitration and predictive network control process of the present invention;

[0027] Figure 3 This is a schematic diagram of the virtualization decoding injection and partitioned display collaborative process of the present invention;

[0028] Figure 4 This is a first-view perspective perspective view of the video conferencing interactive device of the present invention;

[0029] Figure 5 This is a second-view perspective perspective view of the video conferencing interactive device of the present invention.

[0030] The components include: 1. Main body of the device; 2. Display panel; 3. Eye-tracking sensor; 4. Network interface unit. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figure 1 , Figure 4 and Figure 5This invention provides a video conferencing interactive device with adaptive energy-saving control, including a device body 1, a central processing unit, a network interface unit 4, an eye-tracking sensor 3, a video processing unit, a display driver controller, a display panel 2, and a memory.

[0033] The central processing unit establishes an electrical connection and performs data interaction with the network interface unit 4, eye-tracking sensor 3, video processing unit and memory through the system bus. The memory is used to store computer program instructions and buffered data during video conferencing. The network interface unit 4 is used to establish a communication connection with a remote server or multi-point control unit and receive real-time transmission protocol data packets containing multiple video streams.

[0034] An eye-tracking sensor 3 is positioned at a specific location on the front panel of the video conferencing device to collect the user's eye movement data at a preset sampling frequency. Specifically, the eye-tracking sensor 3 is configured to capture the user's gaze point coordinates and eye movement velocity vector. The video processing unit is configured to decode the received video encoded data and output pixel data to the display driver controller. The display driver controller is connected to the display panel 2 and is used to generate timing control signals and driving voltages to control the display panel 2 to display images.

[0035] See attached document Figure 1 The system logic architecture of this invention runs on the above-mentioned hardware environment and specifically includes: a semantic and gaze perception module, an energy consumption arbitration controller module, and a virtualization codec interface layer module.

[0036] The semantic and gaze perception module is connected to the network interface unit 4 and the eye-tracking sensor 3, respectively. The semantic and gaze perception module is configured to perform two parallel tasks:

[0037] First, the audio level indication information and active speaker identification information are parsed from the header extension field of the real-time transmission protocol data packet received from the network interface unit 4.

[0038] Second, it receives raw eye movement data from eye-tracking sensor 3 and calculates the current gaze coordinates P(x,y) and eye movement velocity vector.

[0039] The energy consumption arbitration controller module is connected to the semantic and gaze perception module to receive parsed signaling data and eye-tracking data. The energy consumption arbitration controller module has pre-built comprehensive priority calculation logic for each input video stream S. i The specific logical relationship for calculating the real-time comprehensive priority weight E(i,t) by the energy consumption arbitration controller module is as follows:

[0040] E(i,t)=α·Wsem (i,t)+(1-α)·W gaze (i,t);

[0041] Where i represents the index identifier of the i-th video stream currently being processed; t represents the current calculation time; α represents the preset semantic weight balance coefficient, which ranges from 0 to 1; W sem (i,t) represents the semantic weight value of the i-th video stream at time t, which is determined by the energy consumption arbitration controller module based on audio level indication information and active speaker identification information; W gaze (i,t) represents the gaze weight value of the i-th video stream at time t. This value is determined by the energy consumption arbitration controller module based on the gaze point coordinates P(x,y) and the geometric position relationship between the video stream and the display area on the display panel 2.

[0042] The energy consumption arbitration controller module is further configured to compare the calculated real-time comprehensive priority weight E(i,t) with a preset threshold Γ. When the real-time comprehensive priority weight E(i,t) is greater than or equal to the preset threshold Γ, the energy consumption arbitration controller module generates a first control signal; when the real-time comprehensive priority weight E(i,t) is less than the preset threshold Γ, the energy consumption arbitration controller module generates a second control signal.

[0043] The virtualization codec interface layer module is located between the data output end of the network interface unit 4 and the data input end of the video processing unit. The virtualization codec interface layer module is connected to the power consumption arbitration controller module and is used to receive the first control signal or the second control signal.

[0044] Upon receiving the first control signal, the virtualized codec interface layer module is configured to completely pass through the corresponding video stream data to the video processing unit. Upon receiving the second control signal, the virtualized codec interface layer module is configured to intercept predictive coded frames and bidirectional predictive coded frames in the corresponding video stream data, and generate virtual skip units that are injected into the video processing unit. A virtual skip unit is a network abstraction layer unit that does not contain image residual data; its syntax is constructed as a predictive coded frame in full skip mode.

[0045] The energy consumption arbitration controller module is also connected to the transmitting control logic of network interface unit 4. The energy consumption arbitration controller module is configured to monitor the eye movement velocity vector. The modulus value is set, and when this modulus value exceeds the preset physiological saccade threshold, the energy consumption arbitration controller module determines the power consumption based on the eye movement velocity vector. Calculate the predicted landing area. If the video stream corresponding to the predicted landing area is currently under the control of the second control signal, the power consumption arbitration controller module immediately instructs the network interface unit 4 to send an image loss indication message or an intra-frame request message to the remote server.

[0046] The display driver controller is connected to the energy consumption arbitration controller module. The display driver controller is configured to adjust the refresh rate and backlight current of the corresponding display area on the display panel 2 according to the real-time comprehensive priority weight E(i,t). For the display area corresponding to the second control signal, the display driver controller reduces the refresh rate and backlight current of that area.

[0047] See attached document Figure 2 The semantic and gaze perception module provided by this invention specifically includes: a network protocol parsing submodule and a visual attention calculation submodule.

[0048] The network protocol parsing submodule is connected to the data receive buffer of network interface unit 4. The network protocol parsing submodule is configured to decapsulate the input data packets layer by layer according to the data packet format standard of the Real-Time Transport Protocol (RTP). The network protocol parsing submodule reads the extension bit field of the RTP data packet header. When the extension bit field is set, the network protocol parsing submodule traverses the header extension portion and identifies the audio level indicator field and media stream source identifier field based on the pre-negotiated extension mapping identifier.

[0049] The network protocol parsing submodule extracts the value from the audio level indicator field. This value is represented as an integer between 0 and 127, where a smaller value represents a higher audio energy. The network protocol parsing submodule compares the extracted audio level indicator field value with a preset mute threshold. When the extracted audio level indicator field value is less than the preset mute threshold, the network protocol parsing submodule determines that the participant represented by the media stream source identifier corresponding to the data packet is in an active speaking state.

[0050] The network protocol parsing submodule is further configured to parse the media attribute field in the session description protocol signaling. The network protocol parsing submodule searches for the existence of a content attribute tag in the media attribute field and whether the value of that tag is "slideshow" or "screen". When a content attribute tag with the value "slideshow" or "screen" is detected, the network protocol parsing submodule generates a screen sharing status identifier and associates the screen sharing status identifier with the corresponding media stream source identifier.

[0051] Based on the above parsing results, the network protocol parsing submodule outputs semantic metadata, including media stream source identifier, active speaking status identifier, and screen sharing status identifier, to the energy consumption arbitration controller module.

[0052] The visual attention calculation submodule is connected to the data interface of the eye-tracking sensor 3. The visual attention calculation submodule is configured to receive the raw eye-tracking data stream output from the eye-tracking sensor 3. This raw eye-tracking data stream contains normalized position data of the left and right eyeballs in the sensor coordinate system. The visual attention calculation submodule calls a pre-stored geometric mapping matrix to convert the normalized position data in the sensor coordinate system into gaze point coordinates P(x,y) in the 2-pixel coordinate system of the display panel.

[0053] The visual attention calculation submodule further connects to the window manager interface of the display driver controller or operating system to obtain the layout information of all video conferencing windows on the current screen. The layout information includes the layout information of each video stream S. i The corresponding display area rectangle R i The coordinates of the top left vertex, the width value, and the height value.

[0054] The visual attention calculation submodule executes a geometric inclusion test algorithm to determine whether the gaze point coordinates P(x,y) fall within any display area rectangle R. i Within the range. For the i-th video stream, the visual attention calculation submodule calculates the gaze weight value W based on the result of the geometric inclusion test algorithm. gaze (i,t).

[0055] The visual attention calculation submodule calculates the gaze weight value W. gaze The specific logical relationship of (i,t) is as follows:

[0056]

[0057] Among them, R i d(P,R) represents the rectangular display area of ​​the i-th video stream on display panel 2; i ) represents the distance from the gaze point coordinates P(x,y) to the display area rectangle R. i The shortest Euclidean distance of the boundary; σ represents the visual attenuation parameter, which corresponds to the spatial distribution characteristics of human visual acuity; e represents the base of the natural logarithm.

[0058] When the gaze point coordinates P(x,y) are located within the display area rectangle R i Internally, the visual attention calculation submodule will assign a gaze weight value W. gaze (i,t) is assigned a value of 1.0 when the gaze point coordinates P(x,y) are located within the display area rectangle R. i When external, the visual attention calculation submodule calculates the distance from the gaze point coordinates P(x,y) to the display area rectangle R. i The distance is used to perform exponential decay calculation to obtain the fixation weight value W. gaze(i,t), the visual attention calculation submodule outputs the calculated gaze weight value of each video stream to the energy consumption arbitration controller module.

[0059] The visual attention calculation submodule is also configured to calculate the eye movement velocity vector. The visual attention calculation submodule performs a first-order difference operation on the gaze point coordinates P(x,y) at consecutive sampling times, and divides the difference result by the sampling time interval to obtain the eye movement velocity vector. The visual attention calculation submodule calculates the eye movement velocity vector. The data is transmitted in real time to the energy consumption arbitration controller module for use by subsequent scanning detection logic.

[0060] See attached document Figure 2 The energy consumption arbitration controller module is configured to receive multidimensional input data from the semantic and gaze perception module and perform hierarchical weight calculations to determine the real-time processing strategy for each video stream.

[0061] The energy consumption arbitration controller module first executes the video stream semantic weight W. sem The calculation of (i,t) involves the energy consumption arbitration controller module reading the active speaking status flag and screen sharing status flag corresponding to the i-th video stream, and then using a linear weighted logic model to calculate the semantic weight W of the video stream. sem (i,t), the energy consumption arbitration controller module calculates the semantic weight W of the video stream. sem The specific logical relationship of (i,t) is as follows:

[0062] W sem (i,t)=λ1·S audio (i,t)+λ2·S share (i,t)+λ3;

[0063] Wherein: S audio (i,t) represents the active speaking status indicator value of the i-th video stream at time t. When the participant corresponding to the video stream is in an active speaking state, the energy consumption arbitration controller module will... audio (i,t) is assigned a value of 1; otherwise, it is assigned a value of 0; S share (i,t) represents the screen sharing status indicator value of the i-th video stream at time t. When the content of the video stream is marked as screen sharing content, the power consumption arbitration controller module will... share (i,t) is assigned a value of 1; otherwise, it is assigned a value of 0; λ1 represents the preset gain coefficient for active speaking events; λ2 represents the preset gain coefficient for screen sharing events; λ3 represents the base weight bias coefficient, which is used to ensure that the video stream has a non-zero base weight even when no specific event is triggered.

[0064] The energy consumption arbitration controller module completes the semantic weight W of the video stream. sem After calculating (i,t), the gaze weight value W from the semantic and gaze awareness module is read. gaze The energy consumption arbitration controller module is configured to fuse objective semantic importance with subjective visual attention to calculate the real-time comprehensive priority weight E(i,t) of the i-th video stream. The specific logical relationship of the energy consumption arbitration controller module in calculating the real-time comprehensive priority weight E(i,t) is as follows:

[0065] E(i,t)=α·W sem (i,t)+(1-α)·W gaze (i,t);

[0066] Here, α represents the semantic priority factor, which is a configurable parameter with a value between 0 and 1. It is used to adjust the system's balance preference between the logical importance of meeting content and the importance of user visual behavior. When the semantic priority factor α is set to a large value, the system tends to maintain the high quality of the active speaker's screen, regardless of whether the user is looking at the screen.

[0067] The energy consumption arbitration controller module includes a numerical comparator unit. The energy consumption arbitration controller module inputs the calculated real-time comprehensive priority weight E(i,t) into the numerical comparator unit and compares it with the pre-stored decision threshold Γ.

[0068] When the real-time comprehensive priority weight E(i,t) is greater than or equal to the decision threshold Γ, the energy consumption arbitration controller module determines that the i-th video stream is a critical stream. At this time, the energy consumption arbitration controller module generates and outputs the first control signal to the virtualization codec interface layer module and the display driver controller. The first control signal is used to instruct the subsequent modules to maintain full frame rate decoding and high refresh rate display of the video stream.

[0069] When the real-time comprehensive priority weight E(i,t) is less than the decision threshold Γ, the power consumption arbitration controller module determines that the i-th video stream is a non-critical stream. At this time, the power consumption arbitration controller module generates and outputs a second control signal to the virtualization codec interface layer module and the display driver controller. The second control signal is used to instruct subsequent modules to perform asymmetric decoding and low-power display driving on the video stream.

[0070] To prevent frequent system state switching when the real-time integrated priority weight E(i,t) fluctuates around the decision threshold Γ, the energy consumption arbitration controller module introduces time lag logic in the numerical comparator unit. The energy consumption arbitration controller module only switches the output control signal state when the real-time integrated priority weight E(i,t) continuously meets the decision condition for more than a preset anti-jitter time window (e.g., 200 milliseconds). However, for the process of switching from a non-critical flow to a critical flow, the energy consumption arbitration controller module is configured to bypass the anti-jitter time window to achieve immediate response.

[0071] See attached document Figure 2 The energy consumption arbitration controller module has a dedicated kinematic analysis unit embedded within it, which is configured to process eye movement velocity vectors from the semantic and gaze perception module in real time.

[0072] The energy consumption arbitration controller module first performs velocity magnitude calculation, and then reads the eye movement velocity vector at the current sampling time t. The eye movement velocity vector includes a horizontal component V. x (t) and vertical component V y (t), the energy consumption arbitration controller module calculates the eye movement velocity vector. The Euclidean norm is used to obtain the instantaneous eye movement rate V. mag (t).

[0073] The energy consumption arbitration controller module calculates the instantaneous eye movement rate V. mag The specific logical relationship of (t) is shown below:

[0074]

[0075] Among them, V mag (t) represents the instantaneous eye movement rate at time t; V x (t) represents the horizontal component of the eye movement velocity vector on display panel 2; symbol V y (t) represents the vertical component of the eye movement velocity vector on display panel 2.

[0076] The energy consumption arbitration controller module then performs a physiological threshold comparison, and the energy consumption arbitration controller module pre-stores the saccade judgment threshold ν. sac The threshold for scanning is ν. sac The threshold angular velocity value (e.g., pixel velocity corresponding to 300 degrees / second) is set to characterize the human eye entering a state of saccade inhibition. The energy consumption arbitration controller module will calculate the instantaneous eye movement rate V. mag (t) and the saccadic judgment threshold ν sac Perform numerical comparisons.

[0077] When the instantaneous eye movement rate V mag (t) is greater than the saccadic judgment threshold ν sac At that time, the energy consumption arbitration controller module determines that the current eye movement state is a rapid saccade state.

[0078] To distinguish between unconscious microsaccades and macrosaccades with gaze shifting intent, the energy consumption arbitration controller module is further configured to calculate the eye movement acceleration vector. The energy consumption arbitration controller module performs differential calculations on the eye movement velocity vectors at two consecutive sampling times. When the instantaneous eye movement rate V... mag (t) is greater than the saccadic judgment threshold ν sac And the eye movement acceleration vector Direction and eye movement velocity vector When the directional angle is less than a preset angle threshold (e.g., 30 degrees), the energy consumption arbitration controller module confirms that a valid gaze shift event has been detected.

[0079] Upon confirming the detection of a valid gaze shift event, the energy consumption arbitration controller module immediately marks the system state as saccade suppression. During saccade suppression, the sensitivity of the human visual system to high-frequency spatial information decreases significantly. The energy consumption arbitration controller module uses the state marker as a trigger signal to activate subsequent gaze point prediction and network signaling preemptive transmission logic. The energy consumption arbitration controller module maintains the saccade suppression state marker until the instantaneous eye movement rate V... mag (t) falls back to the saccadic judgment threshold ν sac the following.

[0080] See attached document Figure 2 The energy consumption arbitration controller module immediately initiates the gaze point prediction logic the instant it detects the eye entering a rapid saccade state. The energy consumption arbitration controller module first determines the time span parameter ΔT used for extrapolation calculation. pred The energy consumption arbitration controller module is configured to process the time span parameter ΔT pred Set to a value that matches the average duration of human eye saccades, in one implementation, the energy consumption arbitration controller module determines the value based on the current instantaneous eye movement rate V. mag The magnitude of (t) is determined by retrieving the corresponding time span parameter ΔT from the pre-stored rate-time mapping table. pred .

[0081] The energy consumption arbitration controller module is based on the current gaze point coordinates P(t) and the eye movement velocity vector. Calculate the predicted landing point coordinates P using a kinematic extrapolation model pred The energy consumption arbitration controller module calculates and predicts the landing point coordinates P. pred The specific logical relationship is as follows:

[0082]

[0083] Among them, P pred P(t) represents the coordinates of the predicted gaze point in the 2-pixel coordinate system of the display panel; P(t) represents the coordinates of the gaze point at the moment the saccade detection is triggered. The vector representing the eye movement velocity at the moment the saccade detection is triggered; ΔT represents the eye movement acceleration vector at the moment the saccade detection is triggered. pred This indicates the predicted time span of the pre-defined scanning process.

[0084] In a low-latency implementation, the energy consumption arbitration controller module is configured to ignore the acceleration term in the above formula and only use the velocity vector for linear extrapolation calculation, thereby reducing computational overhead.

[0085] Obtain the predicted landing point coordinates P pred Afterwards, the power consumption arbitration controller module performs a region hit test. The power consumption arbitration controller module obtains the display region layout set {R1,R2,...,R...} of all video streams on the current screen from the display driver controller or system memory. N The display area layout set contains the coordinates of the top-left vertex, width, and height of each video stream display window in the screen coordinate system.

[0086] The energy consumption arbitration controller module determines the predicted landing point coordinates P one by one. pred Is it included in every display area rectangle R? i Within the geometric boundaries.

[0087] When the predicted landing point coordinates P pred The display area rectangle R corresponding to the k-th video stream falls into k When the energy consumption arbitration controller module locks the k-th video stream as the target video stream for scanning, the energy consumption arbitration controller module queries the internal status register to read the current real-time comprehensive priority weight E(k,t) and the currently effective decoding control mode of the target video stream.

[0088] If the target video stream being scanned is currently in a non-critical stream state controlled by the second control signal (i.e., in a low frame rate or virtualization decoding mode), the power consumption arbitration controller module immediately generates a network preemption trigger signal. The power consumption arbitration controller module then transmits the network preemption trigger signal along with the index identifier k of the target video stream being scanned to the subsequent signaling transmission logic, triggering a predictive keyframe request.

[0089] If the predicted landing point coordinates P predIf the video stream does not fall within any valid video stream display area, or if the video stream corresponding to the area it falls within is already in a critical stream state controlled by the first control signal (i.e., full frame rate decoding mode), the power consumption arbitration controller module terminates the current prediction process and does not trigger network signaling operations.

[0090] See attached document Figure 2 The predictive network control mechanism based on saccade suppression period provided by this invention specifically includes signaling construction and transmission timing control logic. After receiving a network preemption trigger signal containing the saccade target video stream index identifier k, the power consumption arbitration controller module immediately establishes communication with the transmission buffer controller of the network interface unit 4. The power consumption arbitration controller module retrieves the synchronization source identifier associated with the k-th video stream. This synchronization source identifier is a 32-bit value used in real-time transmission protocols to uniquely identify the source of the media stream.

[0091] The power consumption arbitration controller module constructs a media stream feedback control message based on the retrieved synchronization source identifier. The media stream feedback control message follows the audiovisual configuration summary standard defined in RFC4585 or RFC5104. Specifically, the power consumption arbitration controller module sets the payload type field of the media stream feedback control message to 206 and the feedback message type field to 1 to form an image loss indication message; or sets the feedback message type field to 4 to form an intra-frame request message.

[0092] The energy consumption arbitration controller module directly writes the constructed media stream feedback control message into the high-priority transmission queue of the network interface unit 4. In response to the write operation, the network interface unit 4 immediately encapsulates the media stream feedback control message into a User Datagram Protocol (UDP) data packet and sends it to the remote media server.

[0093] The time point T at which the energy consumption arbitration controller module executes the above signaling triggering operation trigger Located at the beginning of the user's eye saccade movement, the triggering logic aims to utilize the physiological blind spot time window ΔT of the human visual system during the saccade process. blind To mask the lag in image updates caused by network transmission round-trip latency and video keyframe encoding / decoding latency, the energy consumption arbitration controller module controls the signaling transmission timing to satisfy the following time inequality:

[0094] T trigger +T RTT +T gen +T dec ≤T start +ΔT blind ;

[0095] Among them, T trigger Indicates the time point at which the energy consumption arbitration controller module sends the media stream feedback control message; T RTTThis represents the current average network round-trip time, calculated by network interface unit 4 through periodic sender and receiver reports; T gen This represents the encoding processing time required for the remote media server to generate an instant decoding refresh frame after receiving a request; T dec T represents the processing time required for the video processing unit to decode a real-time decoded refresh frame; start This indicates the absolute time at which the eye-tracking sensor 3 detects the start of the saccade movement; ΔT blind This represents the duration of the human eye in a state of saccade inhibition, and the duration is related to the eye movement velocity vector. It is positively correlated.

[0096] When the above time inequality relationship is satisfied, the video processing unit can complete the decoding of the newly requested high-quality keyframes and update the display buffer before the human eye finishes scanning and refocuses on the target area.

[0097] While sending media stream feedback control messages, the energy consumption arbitration controller module simultaneously updates the status flag bit of the k-th video stream.

[0098] The power consumption arbitration controller module temporarily marks the k-th video stream from a non-critical stream state to a waiting refresh state. During the waiting refresh state, the power consumption arbitration controller module suspends sending the second control signal to the virtualization codec interface layer module and instructs the virtualization codec interface layer module to monitor the input real-time transport protocol data packet stream. When the virtualization codec interface layer module detects the arrival of a new real-time decoding refresh frame, the power consumption arbitration controller module immediately switches the state of the k-th video stream to the critical stream state and resumes the full frame rate decoding mode.

[0099] See attached document Figure 3 The virtualized codec interface layer module provided by the present invention specifically includes: a bitstream syntax analysis submodule, a gated routing submodule, and a buffer queue manager. The virtualized codec interface layer module is located between the jitter buffer output end of the network interface unit 4 and the bitstream input interface of the video processing unit. The virtualized codec interface layer module takes over all video encoded data streams sent to the video processing unit.

[0100] The stream syntax analysis submodule is configured to perform header parsing operations on each input Network Abstraction Layer (NABS) unit. The submodule reads the forbidden bits, reference indicator bits, and unit type field from the NABS unit header. Based on the syntax definition of the video coding standard, the submodule identifies the payload type of the current NABS unit.

[0101] The code stream syntax analysis submodule specifically identifies the following three types of data units:

[0102] The first type is a sequence parameter set or image parameter set unit, which contains global configuration information required for decoding;

[0103] The second type is the instant decoding refresh frame or keyframe unit, which contains independent image reconstruction data;

[0104] The third type is the predictive coded frame or bidirectional predictive coded frame unit, which contains residual data that depends on the reference frame.

[0105] The gated routing submodule is connected to the control signal output of the energy consumption arbitration controller module. The gated routing submodule receives the current control signal status C for the i-th video stream. state (i), the control signal state C state (i) When the value is 0, it corresponds to the first control signal; when the value is 1, it corresponds to the second control signal.

[0106] The gated routing submodule determines the state of the control signal C. state (i) The unit type output by the code stream syntax analysis submodule is used to execute the traffic splitting decision logic. The gating routing submodule directs the input network abstraction layer unit to a direct path or an intercepted path through multiplexer circuits or memory pointer operations. The traffic splitting decision logic function F executed by the gating routing submodule is... route (U p As shown below:

[0107]

[0108] Among them, U k This represents the p-th network abstraction layer unit currently being processed; Type(U k ) represents the syntax type of the k-th network abstraction layer unit; T SPS Indicates the sequence parameter set type identifier; T PPS Indicates the image parameter set type identifier; T IDR Indicates the instant decoding refresh frame type identifier; This indicates a direct path that writes the network abstraction layer unit directly into the input circular buffer of the video processing unit; Pat h intercept This indicates an interception path that discards the load data of the network abstraction layer unit and triggers the subsequent virtual hop unit generation logic; ∈ indicates belonging to. This indicates that it does not belong to this category.

[0109] When the gated routing submodule directs the network abstraction layer unit to the direct path Path direct At this time, the buffer queue manager keeps the write pointer of the video processing unit incremented to ensure that valid data is read by the decoder.

[0110] When the gated routing submodule directs the network abstraction layer unit to the interception path Pathintercept At this time, the buffer queue manager performs a discard operation. The buffer queue manager does not increase the write pointer of the video processing unit, nor does it copy the data of the network abstraction layer unit to the physical memory area of ​​the video processing unit. At the same time, the gated routing submodule sends an injection trigger signal to the virtual unit generator inside the virtualization codec interface layer module.

[0111] Diversion decision logic function F route (U k This ensures that, in low-power mode, although the vast majority of predictive coding frames are physically intercepted to reduce bus bandwidth usage and decoding computation, the parameter set and key reference frames required to maintain the decoding context can always enter the video processing unit without hindrance, thereby avoiding resets or errors in the decoder's internal state machine.

[0112] See attached document Figure 3 The virtualized codec interface layer module contains a virtual unit generation submodule. The virtual unit generation submodule is configured to dynamically synthesize binary bitstream segments that conform to the video encoding standard, i.e., virtual jump units, after receiving the injection trigger signal sent by the gating routing submodule.

[0113] The virtual unit generation submodule first performs header information cloning and adaptation. The virtual unit generation submodule accesses the context status register in the memory to obtain the sequence parameter set ID and image parameter set ID of the current video stream. The virtual unit generation submodule reads the header information of the original network abstraction layer unit that was just intercepted and discarded by the gating routing submodule and extracts the frame number and image display order count from it.

[0114] The Virtual Cell Generation submodule constructs the Network Abstraction Layer (NAL) header of the Virtual Empty Jump Cell. The Virtual Cell Generation submodule sets the Forbidden bit in the NAL header to 0, sets the Reference Indicator bit to a non-zero value, and sets the Cell Type field to the encoded value representing the non-instantaneous decoding refresh slice (e.g., 1 in the H.264 standard).

[0115] The virtual unit generation submodule then constructs the slice header. The virtual unit generation submodule sets the first macroblock address field in the slice header to 0, indicating that the slice covers the entire frame image. The virtual unit generation submodule sets the slice type field in the slice header to a value representing a P slice. The virtual unit generation submodule writes the extracted frame sequence number and image display order count into the corresponding syntax position in the slice header to ensure that the video processing unit considers the timestamps to be continuous during parsing.

[0116] The virtual unit generation submodule performs data padding at the macroblock or coded tree unit level, aiming to generate compressed data indicating full-frame skips without containing any residual coefficients. For the H.264 / AVC coding standard, the virtual unit generation submodule employs the syntax rules of context-adaptive variable-length coding or context-adaptive binary arithmetic coding to generate the macroblock skip count. The virtual unit generation submodule calculates the total number of macroblocks N contained in the current video frame. MB The virtual unit generation submodule constructs a value equal to the total number of macroblocks N. MB The skip execution syntax element instructs the video processing unit to proceed from the first macroblock to N consecutive macroblocks. MB All macroblocks use the skip mode.

[0117] For the H.265 / HEVC encoding standard, the virtual unit generation submodule constructs the syntax flow of the encoding tree unit level, and sets the skip flag of each encoding tree unit to 1.

[0118] Under the skip mode definition, the virtual cell generation submodule implicitly specifies the motion vector and reference frame index. The virtual cell generation submodule forces the specified motion vector prediction difference to be zero vector (0,0) through the above skip syntax elements, and forces the specified reference index in the reference frame list to point to the most recent decoded image frame in the decoded image buffer (i.e., the reference frame with index 0).

[0119] The virtual unit generation submodule performs byte alignment and anti-race code padding at the end. The virtual unit generation submodule adds a stop bit and alignment zero bit to the end of the generated bit stream to meet the byte alignment requirements. The virtual unit generation submodule scans the generated binary data. If it detects a continuous sequence of "0x000x000x01", "0x000x000x02", or "0x000x000x03", it inserts the anti-race byte "0x03" to generate the final virtual skip unit bit stream.

[0120] The virtual unit generation submodule writes the virtual empty jump unit bitstream into the buffer queue manager for the video processing unit to read. Since the virtual empty jump unit is syntactically valid but semantically describes an image that has not changed and has no motion relative to the previous frame, the video processing unit only needs to perform entropy decoding and motion compensation control logic during decoding, without having to perform inverse quantization, inverse transform, and pixel reconstruction operations.

[0121] See attached document Figure 3 The video processing unit integrates a state machine controller, a reference image list manager, a texture reconstruction engine, and a motion compensation engine.

[0122] When the video processing unit reads the virtual skip unit from the buffer queue manager, the video processing unit initiates the state machine spoofing and maintenance process. The state machine spoofing and maintenance process is designed to make the firmware logic of the video processing unit believe that it is currently processing a valid, still video image, thereby maintaining the continuity of the decoding context.

[0123] The state machine controller of the video processing unit first parses the slice header information of the virtual jump unit. The state machine controller extracts the frame number and image display order count, and updates its internal global state variables based on the extracted values. State machine controller updates global state variables The logic is as follows:

[0124] The frame number variable of the currently decoded image Update to the value carried in the virtual jump unit;

[0125] The display order count variable of the currently decoded image. Update to the value carried in the virtual jump unit;

[0126] verify Frame number of the previous frame Does it meet the requirements? The continuity constraint ensures that the verification step will pass because the virtual unit generation submodule has pre-calculated the correct continuity value, thus preventing the video processing unit from triggering frame loss detection or error hiding mechanisms.

[0127] The reference image list manager then performs reference queue maintenance, initializing the reference image list according to the H.264 / H.265 standard protocol. Since the virtual skip unit is marked as a P-slice, the reference image list manager fills in the existing reference frames in the decoded image buffer according to the short-term reference frame indexing rules.

[0128] The reference image list manager performs sliding window memory management operations. If the decoded image buffer is full, the reference image list manager marks the frame that has been stored in the decoded image buffer for the longest time as not to be used for reference, and releases its corresponding physical memory address pointer. It then allocates a logical slot for the currently processed virtual frame, ensuring that the decoder's memory management mechanism can still run normally and will not cause memory overflow, even when virtual frames are injected for a long time.

[0129] At the pixel processing pipeline level, the video processing unit performs bypass power-saving operations. Because the macroblock skip flag or coding unit skip flag in the virtual skip unit is set, the texture reconstruction engine (which usually includes inverse quantization and inverse discrete cosine transform circuits) is turned off by hardware signal gating. The texture reconstruction engine does not perform any matrix operations on the residual coefficients. The motion compensation engine reads the zero motion vector (0,0) and the reference frame index (pointing to the previous frame) in the virtual skip unit. The motion compensation engine does not perform pixel interpolation operations, but enters direct copy mode.

[0130] In direct copy mode, the video processing unit performs zero-copy address mapping. Instead of physically moving the pixel data of the reference frame to the reconstruction buffer of the current frame via the bus, the video processing unit moves the output video memory address pointer of the current frame... Directly points to the reconstructed video memory address of the previous frame The video processing unit establishes the following pointer mapping relationship:

[0131]

[0132] in, This indicates the starting physical address of the current decoded frame in video memory; This indicates the starting physical address of the reference frame (i.e., the previous frame) in video memory; This represents the metadata set of the current decoded frame; This represents the new display timestamp (PTS) calculated based on the virtual jump unit.

[0133] Through the aforementioned pointer mapping, the video processing unit logically generates a new image frame whose content is completely identical to the previous frame, but in reality, it does not consume memory bandwidth for pixel writing. The video processing unit then sends an interrupt signal to the display driver controller, notifying that a new image frame is ready for display. When the display driver controller reads this image, it actually reads the previous frame's data from physical memory that has not been modified, thus achieving seamless stillness of the image.

[0134] See attached document Figure 3 The display driver controller is specifically configured to perform content-based display refresh rate adjustment and region-based backlight power consumption management. Internally, the display driver controller integrates a coordinate mapping logic unit, a multiplexing timing generator, and a light-emitting diode backlight driver array. The display driver controller is connected to the power consumption arbitration controller module and receives each video stream S. i Display area coordinate information R iIn addition to the corresponding control signal states, the display driver controller first activates the coordinate mapping logic unit. The coordinate mapping logic unit stores the physical backlight partition layout parameters of the display panel 2, which divide the display panel 2 into M×N independent backlight control areas Z. m,n , where m is the row index and n is the column index.

[0135] The coordinate mapping logic unit performs geometric intersection operations, calculating the display area coordinates R for each video stream sequentially. i With each backlight control zone Z m,n The overlapping area is determined by the coordinate mapping logic unit. When the overlapping area is greater than a preset coverage threshold (e.g., 50% pixel coverage), the coordinate mapping logic unit establishes the video stream S. i With backlight control area Z m,n The mapping and association relationships.

[0136] The display driver controller then performs backlight current arbitration for each backlight control zone Z. m,n The display driver controller determines the brightness drive coefficient β(m,n) of the area based on the control signal status of the associated video stream. The display driver controller calculates the brightness drive coefficient β(m,n) and the drive current I. drive The logical relationship between (m,n) is as follows:

[0137]

[0138] I drive (m,n)=I max ·L content (m,n)·β(m,n);

[0139] Where β(m,n) represents the brightness driving coefficient of the backlight control area in the m-th row and n-th column; β dim This represents the preset low-power mode brightness attenuation factor, which ranges from 0.1 to 0.5 and is used to reduce backlight power consumption in non-focused areas; I drive (m,n) represents the drive current output to the LED string in the m-th row and n-th column; I max Indicates the maximum rated drive current allowed for display panel 2; L content (m,n) represents the content-adaptive brightness gain calculated based on the local average gray value of the current image content, used to maintain basic contrast.

[0140] The display driver controller transmits the calculated drive current I through a pulse width modulation interface or a constant current source interface. drive(m,n) is applied to the backlight layer of the display panel 2. For video window areas in non-critical flow states, the screen brightness perceived by the human eye will be physically reduced. This, in conjunction with the visual perception attenuation characteristics of the human eye in non-focused areas, achieves hardware-level energy saving without affecting the subjective visual experience.

[0141] Simultaneously, the display driver controller performs partition refresh rate control. The layer compositor inside the display driver controller is configured to independently manage the read frequency of each video stream layer. When video stream S... i When in a non-critical stream state, the display driver controller activates the vertical sync mask logic, which intercepts video memory read requests for that video stream layer during a standard 60Hz or 120Hz screen scan cycle.

[0142] The display driver controller adjusts the frequency reduction ratio K according to the preset ratio. drop (e.g. K) drop =60, meaning it updates only once every 60 frames; the control layer compositor reads the video stream S only during specific scan cycles. i The new data is used to maintain the deflection state of the liquid crystal molecules in the area during the remaining scan cycles. The display driver controller sends a sustain signal to the display panel 2 or uses the panel self-refresh function integrated in the display panel 2.

[0143] Through the above-mentioned coordinated control, when the display driver controller receives a still image (i.e. the decoding result of the virtual jump unit) from the virtualization codec interface layer module, it not only stops updating the screen content logically, but also synchronously reduces the driving current of the backlight in the corresponding area and the switching frequency of the panel driving circuit in the physical circuit, thus realizing asymmetric energy saving across the entire link from the data processing layer to the optoelectronic display layer.

Claims

1. A video conferencing interactive device with adaptive energy-saving control, characterized in that, It includes a central processing unit, a network interface unit, an eye-tracking sensor, a video processing unit, a display driver controller, a display panel, and a memory; The central processing unit is equipped with a semantic and gaze perception module and an energy consumption arbitration controller module. A virtualized codec interface layer module is provided between the data output terminal of the network interface unit and the data input terminal of the video processing unit; The semantic and gaze perception module is configured to parse semantic metadata from the real-time transmission protocol data packets received from the network interface unit, and to obtain gaze point coordinates and eye movement velocity vectors from the eye tracking sensor. The energy consumption arbitration controller module is configured to calculate the real-time comprehensive priority weight for each video stream based on the semantic metadata and the gaze point coordinates, and compare the real-time comprehensive priority weight with a preset threshold. When the real-time comprehensive priority weight is greater than or equal to the preset threshold, a first control signal is generated; When the real-time comprehensive priority weight is less than the preset threshold, a second control signal is generated; The virtualized codec interface layer module is configured to receive the first control signal or the second control signal; when the first control signal is received, the corresponding video stream data is passed through to the video processing unit; when the second control signal is received, the predictive coding frames and bidirectional predictive coding frames in the corresponding video stream data are intercepted, and a virtual jump unit is generated and injected into the video processing unit. The virtual skip unit is a network abstraction layer unit that does not contain image residual data, and the virtual skip unit is constructed in a full skip mode; The display driver controller is configured to reduce the refresh rate and backlight current of the display area corresponding to the second control signal, based on the real-time comprehensive priority weight.

2. The video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The energy consumption arbitration controller module is configured to calculate a real-time comprehensive priority weight for each video stream based on the semantic metadata and the gaze point coordinates, specifically including: Obtain the semantic weight balance coefficient and determine the semantic weight value based on the audio level indication information and the active speaker identification information; The gaze weight value is determined based on the geometric positional relationship between the gaze point coordinates and the display area rectangle of the video stream on the display panel; The real-time comprehensive priority weight is obtained by weighted summation of the semantic weight value and the gaze weight value. When the gaze point coordinates are located inside the display area rectangle, the gaze weight value is the maximum normalized value; when the gaze point coordinates are located outside the display area rectangle, the gaze weight value is calculated by exponential decay based on the Euclidean distance from the gaze point coordinates to the display area rectangle.

3. A video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The energy consumption arbitration controller module is also configured to monitor the magnitude of the eye movement velocity vector; When the magnitude of the eye movement velocity vector exceeds the preset physiological saccade threshold, the energy consumption arbitration controller module determines that it has entered the saccade inhibition period and calculates the predicted landing area based on the eye movement velocity vector. If the video stream corresponding to the predicted landing point area is currently under the control of the second control signal, the power consumption arbitration controller module generates a network preemption trigger signal and instructs the network interface unit to send a media stream feedback control message to the remote server. The media stream feedback control message is either an image loss indication message or an intra-frame request message.

4. A video conferencing interactive device with adaptive energy-saving control according to claim 3, characterized in that, The specific steps of calculating and predicting the landing point region based on the eye movement velocity vector include: Obtain the gaze coordinates, eye movement velocity vector, and eye movement acceleration vector at the current sampling time, and determine the predicted time span of the saccade process; Based on the gaze point coordinates, the eye movement velocity vector, and the eye movement acceleration vector, the coordinates of the predicted landing point region in the pixel coordinate system of the display panel are calculated using a kinematic extrapolation model; The timing of the media stream feedback control message sent by the energy consumption arbitration controller module is configured to satisfy the following timing constraints: the sum of the trigger time, network round-trip delay, remote encoding processing time, and local decoding processing time is less than or equal to the sum of the start time of the saccade action and the duration of the saccade suppression state.

5. A video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The virtualization codec interface layer module includes a gated routing submodule; The gated routing submodule is configured to perform header parsing on the input network abstraction layer unit to identify the sequence parameter set, image parameter set, instantaneous decoding refresh frame and the predictive encoded frame. When the second control signal is received, the gated routing submodule directs the sequence parameter set, the image parameter set, and the instant decoding refresh frame to the pass-through path into the video processing unit, and directs the predictive encoded frame to the intercept path for discarding, while triggering the generation logic of the virtual jump unit.

6. A video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The virtualized codec interface layer module further includes a virtual unit generation submodule, which is configured to construct the virtual jump unit. The construction steps of the virtual unit generation submodule include: Extract the frame sequence number and image display order count of the intercepted original network abstraction layer unit, and write the extracted frame sequence number and image display order count into the slice header of the virtual jump unit; Construct the syntax structure for the full skip mode, set the number of macroblock skips in the virtual skip unit to the total number of macroblocks in the current video frame, or set the skip flag of the coding tree unit to the valid state; The syntax structure of the full skip mode instructs the video processing unit to decode using zero motion vectors and a reference frame index pointing to the previous frame.

7. A video conferencing interactive device with adaptive energy-saving control according to claim 6, characterized in that, The video processing unit is configured to perform a zero-copy address mapping operation in response to the virtual skip unit; The zero-copy address mapping operation includes: directly pointing the output memory address pointer of the current decoded frame to the physical memory address of the reconstructed image of the previous frame, bypassing the inverse quantization and inverse transform operations of the texture reconstruction engine inside the video processing unit, and only updating the global state variables inside the decoder to maintain the continuity of the decoding context.

8. A video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The display driver controller integrates a coordinate mapping logic unit. The coordinate mapping logic unit stores the physical backlight partition layout parameters of the display panel. The coordinate mapping logic unit is configured to calculate the overlap area between the display area of ​​the video stream and the backlight control area. When the overlap area exceeds a preset coverage ratio, a mapping relationship between the video stream and the backlight control area is established. According to the mapping relationship, the display driver controller outputs a reduced driving current for the backlight control area associated with the video stream that is in the control state of the second control signal.

9. A video conferencing interactive device with adaptive energy-saving control according to claim 8, characterized in that, The reduced output drive current specifically includes: Obtain the maximum rated drive current, content-adaptive brightness gain, and brightness drive coefficient; The driving current is obtained by multiplying the maximum rated driving current, the content adaptive brightness gain, and the brightness driving coefficient. When the video stream associated with the backlight control area is in a state controlled by the second control signal, the brightness driving coefficient is set to a preset low-power mode brightness attenuation factor.

10. A video conferencing interactive device with adaptive energy-saving control according to claim 1, characterized in that, The display driver controller is configured to activate the vertical synchronization mask logic; For a video stream that is under the control of the second control signal, the display driver controller intercepts video memory read requests for the video stream layer according to a preset down-rate ratio, reads new data of the video stream only in scan cycles that meet the down-rate ratio, and keeps the liquid crystal molecules of the display panel unchanged in the remaining scan cycles.

Images