Video packet loss synchronization compensation method and system based on deep learning

Abstract

The application discloses a video packet loss synchronization compensation method and system based on deep learning, which comprises the following steps: obtaining a video stream to be compensated and extracting original video frame features, and loading the original video frame features into a packet loss compensation model comprising an LSTM prediction network and a GAN generator network, wherein the time series prediction unit comprises a long short-term memory component with an attention gate; the LSTM prediction network sequentially performs inter-frame motion mapping, region enhancement mapping and region feature fusion on the original video frame features through the time series prediction unit to obtain a predicted feature sequence; the GAN generator generates decoding target substitute frame features from the predicted feature sequence, and then obtains a synchronization compensation frame from the video stream to be compensated. The method adjusts the time series state by means of the attention gate, realizes the cooperation between time series prediction and detail generation by combining LSTM and GAN, improves the time series continuity and detail accuracy of the compensation frame, and effectively solves the problem of synchronization compensation in the video packet loss scenario.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a video packet loss synchronization compensation method and system based on deep learning. Background Technology

[0002] During video transmission, packet loss is common due to network bandwidth fluctuations and transmission delays, causing stuttering and image distortion at the receiving end, severely impacting the user's viewing experience. Existing video packet loss compensation methods often rely on frame duplication or simple interpolation, which struggles to restore the temporal continuity and detailed features of lost frames. While some deep learning-based methods incorporate temporal prediction networks, they suffer from inaccurate temporal state adjustment and insufficient regional feature fusion, resulting in poor synchronization between the compensated frames and the original video stream, leading to blurry images. Summary of the Invention

[0003] The purpose of this invention is to provide a video packet loss synchronization compensation method and system based on deep learning.

[0004] In a first aspect, embodiments of the present invention provide a video packet loss synchronization compensation method based on deep learning, comprising:

[0005] The video stream to be compensated is acquired, and frame features are extracted from the video stream to obtain the original video frame features. The original video frame features are then loaded into the packet loss compensation model. The packet loss compensation model includes an LSTM prediction network and a GAN generator network set in sequence. Both the LSTM prediction network and the GAN generator network contain multiple sequentially connected temporal prediction units. Each temporal prediction unit includes a long short-term memory component. The long short-term memory component is equipped with an attention gating mechanism. The attention gating mechanism is used to guide the adjustment of the internal temporal state of the long short-term memory component or to maintain the internal temporal state constant.

[0006] Based on the first temporal prediction unit in the LSTM prediction network, inter-frame motion mapping is performed on the original video frame features to obtain a first temporal feature, region enhancement mapping is performed on the first temporal feature to obtain a second temporal feature, and region feature fusion is performed on the second temporal feature to obtain the output of the first temporal prediction unit. The output of the first temporal prediction unit is then loaded into the next temporal prediction unit until the prediction feature sequence of the LSTM prediction network is obtained.

[0007] Based on the GAN generator network, the predicted feature sequence of the LSTM prediction network is generated and decoded to obtain the target replacement frame features;

[0008] Based on the target replacement frame features, frames are synchronously generated from the video stream to be compensated to obtain synchronous compensation frames.

[0009] Secondly, embodiments of the present invention provide a server system, including a server, the server being used in the method described in the first aspect.

[0010] Compared to existing technologies, the beneficial effects provided by this invention include: The method and system for video packet loss synchronization compensation based on deep learning, as disclosed in this invention, involves acquiring the video stream to be compensated and extracting the features of the original video frames. These features are then loaded into a packet loss compensation model containing an LSTM prediction network and a GAN generator network. The temporal prediction unit includes a long short-term memory component with attention gating. The LSTM prediction network sequentially performs inter-frame motion mapping, region enhancement mapping, and region feature fusion on the original video frame features through the temporal prediction unit to obtain a predicted feature sequence. The GAN generator decodes the predicted feature sequence to obtain target replacement frame features, thereby obtaining a synchronization compensation frame from the video stream to be compensated. This method guides temporal state adjustment through attention gating and combines LSTM and GAN to achieve synergy between temporal prediction and detail generation, improving the temporal coherence and detail accuracy of the compensation frame, and effectively solving the synchronization compensation problem in video packet loss scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the steps of the deep learning-based video packet loss synchronization compensation method provided in an embodiment of the present invention.

[0013] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0015] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0016] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the deep learning-based video packet loss synchronization compensation method provided in this embodiment. The deep learning-based video packet loss synchronization compensation method will be described in detail below.

[0017] Step S201: Obtain the video stream to be compensated, extract frame features from the video stream to be compensated to obtain original video frame features, and load the original video frame features into the packet loss compensation model. The packet loss compensation model includes an LSTM prediction network and a GAN generator network set in sequence. Both the LSTM prediction network and the GAN generator network contain multiple sequentially connected temporal prediction units. The temporal prediction unit includes a long short-term memory component. The long short-term memory component is equipped with an attention gating. The attention gating is used to guide the adjustment of the internal temporal state of the long short-term memory component or to maintain the internal temporal state constant.

[0018] Step S202: Based on the first temporal prediction unit in the LSTM prediction network, perform inter-frame motion mapping on the original video frame features to obtain a first temporal feature, perform region enhancement mapping on the first temporal feature to obtain a second temporal feature, perform region feature fusion on the second temporal feature to obtain the output of the first temporal prediction unit, and load the output of the first temporal prediction unit into the next temporal prediction unit until the prediction feature sequence of the LSTM prediction network is obtained;

[0019] Step S203: Generate and decode the predicted feature sequence of the LSTM prediction network based on the GAN generator network to obtain the target replacement frame features;

[0020] Step S204: Based on the target replacement frame features, a frame is synchronously generated from the video stream to be compensated to obtain a synchronous compensation frame.

[0021] In this embodiment of the invention, exemplarily, a real-time video conferencing system is used as the application scenario. The server, as the execution entity for video stream processing and compensation, receives real-time video streams from the client (using TCP to transmit keyframes and UDP to transmit non-keyframes). When UDP packet loss is detected, resulting in the loss of non-keyframes, a replacement frame is generated using the deep learning-based video packet loss synchronization compensation method of this invention to ensure the smoothness and synchronization accuracy of the received video. The following describes each step of the method in detail with reference to a specific scenario.

[0022] The server first receives the video stream sent by the client via Real-Time Transport Protocol (RTP). In this video stream, key frames (such as I-frames, which contain complete image information) are transmitted via TCP to ensure reliability, while non-key frames (such as P-frames and B-frames, which are based on differential information from preceding frames) are transmitted via UDP to reduce latency. The server has a built-in packet loss detection module that monitors in real time whether non-key frames transmitted via UDP are lost by parsing the sequence number field of the RTP packets. For example, in a video conferencing scenario, client A sends a video stream to client B. The server detects that a P-frame (non-key frame) with RTP sequence number 1001 did not arrive within a preset time window (such as 50ms), and that the adjacent frames with sequence numbers 1000 (P-frame) and 1002 (B-frame) are lost. The server then determines that the P-frame is lost and triggers the packet loss compensation process.

[0023] The server identifies the valid video frames before and after packet loss (i.e., P-frame 1000 and B-frame 1002) as the "video stream to be compensated" and extracts frame features from them. Specifically, the server calls a pre-trained convolutional neural network (such as the first 10 layers of ResNet-50) to extract features from the two frames: first, each frame is resized to 256×256 resolution, and low-level visual features (such as edges, textures, and color channel distribution) are extracted through convolutional layers (3×3 kernels, stride 1, padding 1). Then, dimensionality is reduced through pooling layers (2×2 max pooling), and the final output is a feature map with a dimension of 64×64×128 (containing spatial domain details and temporal domain motion trends), denoted as "original video frame features". For example, in a video conference scene where a person is speaking, the original video frame features will focus on capturing the texture features of the person's face (such as the outline of the eyes and mouth), the motion vectors of the upper body (such as the direction of hand gestures), and the static features of the background area (such as the edges of the conference room tables and chairs).

[0024] Subsequently, the server loads the original video frame features into a pre-trained "packet loss compensation model." This model comprises a sequentially connected LSTM prediction network and a GAN generator network: the LSTM prediction network predicts the temporal feature sequence of the lost frame based on the features of the preceding and following frames, while the GAN generator network decodes the predicted feature sequence into directly displayable replacement frames. The LSTM prediction network consists of three cascaded "temporal prediction units," and the GAN generator network consists of four cascaded temporal prediction units. Each temporal prediction unit contains a "long short-term memory component" and an "attention gating." The attention gating dynamically adjusts the internal temporal state of the long short-term memory component based on the video content—for example, maintaining the stability of the temporal state in the facial region (a high-information region) to preserve details, and adjusting the temporal state in the background region (a low-information region) to reduce redundant computation.

[0025] The core function of the LSTM prediction network is to output a "predicted feature sequence" of the lost frame based on the original video frame features (features of frames before and after packet loss) through the cascading processing of multiple temporal prediction units. The following uses the first temporal prediction unit of the LSTM prediction network as an example to explain its processing in detail. The processing logic of subsequent temporal prediction units is the same, only the parameters and feature dimensions are dynamically adjusted with the network depth.

[0026] The server first performs "inter-frame motion mapping" on the original video frame features using the Long Short-Term Memory (LSTM) component of the first temporal prediction unit to capture the motion correlation between consecutive frames, outputting the "first temporal feature". The specific steps are as follows: Standardization: The server performs Z-score standardization on the original video frame features (64×64×128) using the formula... The eigenvalues ​​are adjusted to a distribution with a mean of 0 and a variance of 1, where The global mean of the feature map. The standard deviation is the global standard deviation. Standardization aims to eliminate the interference of differences in the dimensions of different feature channels on subsequent motion mapping. For example, the texture feature value range (0-255) of the facial region and the brightness feature value range (0-100) of the background region need to be standardized. Motion mapping of the Long Short-Term Memory (LSTM) unit: The standardized features are input into the LSTM unit, which includes an input gate, a forget gate, an output gate, and cell states to learn temporal dependencies between frames. In video conferencing scenarios, inter-frame motion is mainly manifested as head rotation and hand gestures. The LSTM unit filters irrelevant historical information (such as a water glass briefly appearing in a previous frame) through the forget gate, retains key motion features (such as the displacement vector of the face) through the input gate, and updates the cell states. At this point, attention gating intervenes to adjust the internal temporal state of the LSTM unit. The server dynamically calculates the "regional attention confidence" based on the regional importance of the video content: For the current processing stage (the first processing cycle of the temporal prediction unit), the server obtains the "initial regional attention confidence" (e.g., 0.8 for the face region, 0.2 for the background region) and the "initial regional attention confidence fluctuation" (e.g., 0.05 for the face region, 0.15 for the background region) from the previous temporal stage (the feature processing stage of frame 1000 before packet loss) using formula C. target =C initThe "target region attention confidence" is calculated using +α·△C (where α is an adjustment coefficient, dynamically adjusted based on the intensity of video motion; α=0.3 when the person turns their head quickly, and α=0.1 in a static scene). For example, when the person's head turns from left to right, the target attention confidence of the facial region is updated to 0.8 + 0.3 × 0.05 = 0.815, and the background region is updated to 0.2 + 0.3 × 0.15 = 0.245. This means that attention gating guides the Long Short-Term Memory (LSTM) component to prioritize maintaining the temporal stability of the facial region while appropriately paying attention to changes in the background region. Temporal dimension integration: The LSM component outputs a feature map of a single temporal step (64×64×128). The server performs a temporal dimension integration operation (average pooling along the time axis) to fuse the motion features of the two frames before and after packet loss into a "first temporal feature" (dimension 64×64×256). This feature not only includes spatial details of a single frame, but also integrates motion trends between two frames (e.g., the displacement of a person's face in the x-axis direction is 5 pixels / frame, and the displacement in the y-axis direction is 0 pixels / frame).

[0027] To further highlight the features of key regions in the video (such as faces and hands), the server performs "region enhancement mapping" on the first temporal feature, outputting "second temporal features". The specific process is as follows: Spatial filtering and local relation segmentation: The server first performs a 3×3 convolutional filter on the first temporal feature (64×64×256) in the spatial dimension based on the temporal dimension (the convolutional kernel weights are determined through pre-training, and the bias term is 0), obtaining a "second feature map" (64×64×256) to smooth noise and enhance regional correlation. Subsequently, the second feature map is standardized (LayerNorm, calculating the mean and variance along the channel dimension) and divided into "first local relation features" and "second local relation features" based on the spatial dimension. For example, the 64×64 feature map is divided into 16 local regions through a sliding window (16×16 pixels), including 4 regions of the face as "first local relation features" (high priority regions) and the remaining 12 background regions as "second local relation features" (low priority regions). Local region feature enhancement: The server further standardizes the first local relational feature (high-priority region) and performs a 3×3 convolutional filter on the local region dimension based on the temporal dimension (the convolutional kernel is optimized for facial features, such as enhancing edge response), resulting in the "first feature map" (16×16×256, corresponding to refined features of the facial region). Subsequently, the element-wise multiplication result of the first feature map and the second local relational feature (background region) is calculated (highlighting the correlation between the face and the background), and then added to the second local relational feature (preserving basic background information). Finally, it is integrated through a 3×3 convolutional filter to obtain the "second temporal feature" (64×64×256). For example, in a scene of a person raising their hand in a gesture, the second temporal feature will enhance the motion vector features of the hand region (such as the wrist rotation angle) while weakening the interference of wall textures in the background.

[0028] The server performs "regional feature fusion" on the second temporal features, dynamically adjusting the weights of different regions through a feature selection mechanism, and outputs the processing result of the current temporal prediction unit. The specific steps are as follows: Parallel spatial filtering and feature selection: The server first standardizes the second temporal features (BatchNorm, normalization along the batch dimension), and then performs spatial filtering through two parallel convolutional layers: the first convolutional layer uses a 3×3 kernel (stride 1, padding 1), outputting a "third feature map" (64×64×128, containing global semantic features); the second convolutional layer uses a 1×1 kernel, outputting a "fourth feature map" (64×64×128, used to generate feature selection weights). The server scales the fourth feature map using the Sigmoid function (output range 0-1) to obtain "feature selection weights"—for example, the weight for the face region is 0.85, the weight for the hand region is 0.7, and the weight for the background region is 0.2, meaning that features in high-information regions are preferentially retained through weight allocation. Weighted fusion output: The server multiplies the third feature map element-wise with the feature selection weights (giving stronger representation to features in high-weight regions) to obtain the output features of the current temporal prediction unit (64×64×128). This output is loaded into the next temporal prediction unit of the LSTM prediction network (3 in series), repeating the above process of "inter-frame motion mapping → region enhancement mapping → region feature fusion", finally outputting a "predicted feature sequence" (dimension 64×64×512), which contains the temporal features and spatial details of the lost frame (P-frame with sequence number 1001).

[0029] The predicted feature sequence output by the LSTM prediction network is input into the GAN generator network. This network transforms the abstract feature sequence into directly displayable image features (i.e., "target surrogate frame features") through a "generative decoding" process. The GAN generator network consists of four cascaded temporal prediction units (with the same structure as the units in the LSTM prediction network) and a transposed convolutional layer. Its core is to optimize the visual realism of the generated frames through adversarial training—the generator network generates surrogate frame features, and the discriminator network (jointly trained during pre-training) distinguishes between the generated features and the real frame features. Both networks dynamically adjust their parameters using the minimax loss function.

[0030] In video conferencing scenarios, the generator network first upsamples the predicted feature sequence (64×64×512): it gradually restores the feature map resolution through four layers of transposed convolutions (each with a 4×4 kernel, stride of 2, and padding of 1), from 64×64 to 128×128 to 256×256, while the number of channels changes from 512 to 256 to 128 to 3 (corresponding to three RGB channels). For example, the transposed convolutional layer learns the skin color distribution of a person's face in a real frame (such as the Cr component range in the YCrCb color space) to ensure that the facial region color in the generated replacement frame is natural; by capturing the continuity of gesture movements (such as the intermediate posture from "raising a hand" to "lowering a hand"), it ensures that the generated frame has the same motion trend as the preceding and following frames.

[0031] The generator network also incorporates feedback from a "frame quality fluctuation detection unit" during the decoding process. This unit calculates in real time the spatial domain detail deviation (such as edge sharpness difference), temporal domain detail deviation (such as motion vector continuity error), and spectral envelope feature deviation (such as blurring caused by high-frequency component loss) between the generated features and the real frame features. It then optimizes the generator parameters through backpropagation using loss functions (such as perceptual loss and GAN loss) and finally outputs the "target replacement frame features" (256×256×3), which are the image features of the lost frames.

[0032] Based on the characteristics of the target replacement frame, the server extracts synchronization information (such as RTP timestamps and motion vector synchronization parameters) from the video stream to be compensated. It then aligns the generated replacement frame with the preceding and following frames using timestamps and matches their motion vectors to ultimately obtain the "synchronization compensation frame." Specifically, the server reads the timestamps (e.g., 1620000000ms) of the frame before packet loss (frame 1000) and the timestamp of the frame after packet loss (frame 1002) (1620000033ms) from the video stream to be compensated. Based on the video frame rate (30fps, 33ms interval per frame), it calculates the target timestamp of the lost frame (frame 1001) as 1620000016.5ms. The timestamp of the generated replacement frame is set to this value, ensuring that the time interval error with the preceding and following frames is ≤15ms (meeting synchronization accuracy requirements).

[0033] Simultaneously, the server uses a motion vector synchronization module to match the motion vectors of the replacement frame with those of the preceding and following frames. For example, if the motion vector of the person's face in the preceding frame 1000 is (+5px, 0px) (shifted 5 pixels to the right), and the motion vector in the following frame 1002 is (+10px, 0px), then the motion vector of the replacement frame 1001 is calibrated to (+7.5px, 0px), ensuring continuous motion trajectories. Finally, the server sends the video stream containing the synchronization compensation frames to the receiving client B via the RTP protocol. Client B's player decodes and plays the video in timestamp order, achieving smooth video playback.

[0034] In this embodiment, the server successfully generates replacement frames with high visual consistency and temporal synchronization with the original frames in video conferencing scenarios with UDP packet loss through the entire process of "acquiring the video stream to be compensated → extracting original frame features → LSTM prediction network temporal feature prediction → GAN generator network decoding → synchronous compensation frame output". This effectively solves the problem of screen tearing and stuttering caused by packet loss.

[0035] In this embodiment of the invention, the step of obtaining the first temporal feature by performing inter-frame motion mapping on the original video frame features can be implemented through the following example: The original video frame features are standardized, and inter-frame motion mapping is performed on the standardized original video frame features using the long short-term memory component;

[0036] The output of the long short-term memory component is integrated along a time-series dimension to obtain the first time-series feature.

[0037] In this embodiment of the invention, for example, in a video conferencing scenario, after the server detects the loss of P-frame with sequence number 1001 in UDP transmission, it acquires P-frame number 1000 and B-frame number 1002 before and after the packet loss as the video stream to be compensated. Frame feature extraction is performed on these two frames to obtain original video frame features of 64×64×128, including facial texture, upper body motion vectors, and background static features. The server first performs Z-score normalization on the original video frame features, adjusting the feature values ​​to a distribution with a mean of 0 and a variance of 1, to eliminate the dimensional differences between facial texture features (value range 0-255) and background brightness features (value range 0-100). Next, the normalized features are input into a Long Short-Term Memory (LSTM) component. This component filters out irrelevant historical information such as a water cup that briefly appears in previous frames through a forget gate, retains key motion features such as facial displacement vectors through an input gate, and updates cell states. During this process, the attention gating dynamically adjusts its internal temporal state based on the importance of the region. For example, when the character's head rotates, it prioritizes maintaining the stability of the facial region's temporal state while also paying appropriate attention to changes in the background region. Subsequently, the server performs average pooling along the time axis on the single temporal step feature map (64×64×128) output by the Long Short-Term Memory component, fusing the motion features of the two frames before and after packet loss to obtain a first temporal feature with a dimension of 64×64×256. This feature integrates the spatial details of a single frame with the motion trend between two frames, such as the displacement of the character's face in the x-axis direction of 5 pixels / frame.

[0038] In this embodiment of the invention, the packet loss compensation model is configured to perform cyclic processing through multiple time-series stages. The step of performing inter-frame motion mapping on the standardized original video frame features through the long short-term memory component can be implemented through the following example.

[0039] For any current stage among the multiple time-series stages, obtain the initial region attention confidence and initial region attention confidence fluctuation of the long short-term memory component in the previous time-series stage of the current stage, and determine the target region attention confidence of the long short-term memory component in the current stage based on the initial region attention confidence and the initial region attention confidence fluctuation.

[0040] The attention confidence of the target region is quantified to obtain the attention weight corresponding to the attention gating in the current stage.

[0041] The internal temporal state of the Long Short-Term Memory (LSTM) component in the current stage is determined based on the attention weights, and inter-frame motion mapping is performed on the standardized original video frame features through the LSM component based on the internal temporal state.

[0042] In an embodiment of the invention, for example, in a video conferencing scenario, user A is speaking and making hand gestures. The server detects the loss of UDP transmission frame 2003 (corresponding to the intermediate gesture of A raising his hand), triggering a packet loss compensation process. The packet loss compensation model sets three time-series stages (T1, T2, T3) to cyclically process the video frame features before and after packet loss (frame 2002: A looks down at the document; frame 2004: A raises his hand to his chest) to gradually refine the motion features. The following focuses on the current time-series stage T2 (which follows the preliminary motion analysis of stage T1 and transitions to the fine motion prediction of stage T3), detailing the inter-frame motion mapping process of the Long Short-Term Memory component. The server first obtains the "initial region attention confidence" and "initial region attention confidence fluctuation" at the end of the previous time-series stage T1. In phase T1, feature analysis of frame 2002 identified high-information regions: the facial region (A's facial features, including expressions and head-raising movements) had an initial confidence score of 0.85 (highest attention); the hand region (A's right hand, just raised from the table) had an initial confidence score of 0.6 (second highest attention); and the background region (meeting room tables, chairs, and whiteboard) had an initial confidence score of 0.2 (lowest attention). The initial region attention confidence fluctuation reflects the magnitude of attention changes for each region: the facial region, due to the head-raising movement, tends to be stable, with a fluctuation of 0.03 (small fluctuation); the hand region, due to the initial raising movement, has more vigorous motion, with a fluctuation of 0.12 (medium fluctuation); and the background region shows no significant change, with a fluctuation of 0.05 (micro-fluctuation). The server dynamically adjusts the coefficient α based on the intensity of motion in the video content: for the stable facial region, α is set to 0.1 (weakening the impact of fluctuations); for the vigorous hand region, α is set to 0.4 (strengthening the response to fluctuations); and for the background region, with minimal changes, α is set to 0.2 (balancing stability and update requirements). The target confidence level for each region in stage T2 is calculated using the formula: C_target = initial confidence level C_init + α × fluctuation ΔC. For the face region: 0.85 + 0.1 × 0.03 = 0.853 (slight increase in attention, remaining stable); for the hand region: 0.6 + 0.4 × 0.12 = 0.648 (significant increase in attention, adapting to accelerated movement); for the background region: 0.2 + 0.2 × 0.05 = 0.21 (small increase in attention, capturing subtle background changes). The server quantifies the target region attention confidence level, normalizing each region's target confidence level to a 0-1 "attention weight" (with a total weight of 1) using the Softmax function, thus clarifying the attention priority of the Long Short-Term Memory (LSTM) component for different regions.The specific calculations are as follows: Total confidence score for each region = 0.853 (face) + 0.648 (hands) + 0.21 (background) = 1.711; Attention weight for the face region: 0.853 / 1.711 ≈ 0.5 (50% of the total, core focus); Attention weight for the hands region: 0.648 / 1.711 ≈ 0.38 (38% of the total, key focus); Attention weight for the background region: 0.21 / 1.711 ≈ 0.12 (12% of the total, secondary focus). The internal temporal states of the long short-term memory components include cellular states (long-term memory, storing historical motion features) and hidden states (short-term output, reflecting current motion trends). The server dynamically adjusts these two states based on attention weights: Facial region (weight 0.5): Due to the highest attention and stable movement, the cell state retains 80% of the historical information from stage T1 (such as A's facial contour and eye opening / closing), and only updates 20% of the new features (chin angle change when raising the head), ensuring the continuity of expression and head posture; the hidden state focuses on outputting the vertical displacement vector of the face (displacement rate from looking down to looking up: 1.2 pixels / time stage). Hand region (weight 0.38): Due to the intense movement and secondary attention, the cell state updates 60% of the historical information from stage T1 (such as the initial position coordinates of the right hand), retains 40% of the core information (palm orientation), to adapt to changes in the amount of hand movement; the hidden state outputs the composite motion vector of the hand in the horizontal and vertical directions (horizontal displacement to the right 2.5 pixels / time stage, vertical displacement upward 3 pixels / time stage, matching the trajectory of raising the hand to the chest). Background region (weight 0.12): Due to low attention, the cell state retains only 30% of the historical information from stage T1 (such as the texture of the whiteboard edge), quickly updates 70% of the redundant information (filtering out minor fluctuations in desktop reflections), and the hidden state weakens spatial details, outputting smooth static features. Based on the above internal temporal states, the Long Short-Term Memory (LSTM) component performs inter-frame motion mapping on the standardized original video frame features (including facial, hand, and background features of frame 2002): the cell state anchors the historical motion baseline, the hidden state captures the current motion trend, and finally outputs a motion feature map (64×64×128) with fused region priority, completing the inter-frame motion mapping in stage T2. This feature map will serve as the input for the region enhancement mapping in stage T3, further refining the motion details of the lost frames.

[0043] In this embodiment of the invention, after quantifying the confidence level of the target region to obtain the attention weight corresponding to the attention gating in the current stage, the method also provides the following implementation methods.

[0044] Based on the attention weight corresponding to the current stage, determine the initial region attention confidence level corresponding to the current stage;

[0045] Based on the attention weight corresponding to the current stage and the target region attention confidence, the initial region attention confidence fluctuation corresponding to the current stage is determined.

[0046] In this embodiment of the invention, exemplarily, in a video conferencing scenario, after the server completes the attention weight calculation for stage T2 (face 0.5, hands 0.38, background 0.12), it needs to determine the initial region attention confidence and fluctuation when this stage is considered the "previous time series stage" of the next time series stage T3, providing a basis for motion mapping in stage T3. The server converts the attention weights of stage T2 into initial confidence through linear mapping: using the highest historical initial confidence of 0.85 corresponding to the maximum weight (face 0.5) as a benchmark, a scaling factor of 1.7 is calculated (0.85 / 0.5=1.7). Initial confidence for each region = attention weight × 1.7: face region 0.5 × 1.7 = 0.85 (maintaining core attention), hand region 0.38 × 1.7 = 0.646 (matching the need for raising hand movements), background region 0.12 × 1.7 = 0.204 (retaining basic static attention). These values ​​will serve as the "initial region attention confidence" for stage T3. The server calculates the fluctuation based on the difference between the target region attention confidence score in stage T2 (face 0.853, hands 0.648, background 0.21) and the initial confidence score in the current stage (face 0.85, hands 0.646, background 0.204), combined with attention weights: Fluctuation = (Target Confidence - Initial Confidence) × Attention Weight. For the face region, (0.853 - 0.85) × 0.5 = 0.0015 (stable movement, minimal fluctuation); for the hands region, (0.648 - 0.646) × 0.38 = 0.00076 (hand movement stabilizing, fluctuation narrowing); and for the background region, (0.21 - 0.204) × 0.12 = 0.00072 (primarily static, extremely small fluctuation). These values ​​will be used as the "Initial Region Attention Confidence Fluctuation" for stage T3.

[0047] In this embodiment of the invention, the packet loss compensation model further includes a frame quality fluctuation detection unit, and the following implementation method is provided before loading the original video frame features into the packet loss compensation model.

[0048] Obtain a lost video stream instance and a corresponding reference video stream instance, and call the packet loss compensation model to generate packet loss compensation for the lost video stream instance;

[0049] The frame detail fluctuation of the compensation frame generation result of the packet loss video stream instance is determined by the frame quality fluctuation detection unit. The training error value is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. The packet loss compensation model is trained based on the training error value.

[0050] In this embodiment of the invention, for example, during the model training phase of a video conferencing scenario, the server needs to optimize the packet loss compensation model parameters using historical data to ensure that the compensation frames generated in actual applications are highly consistent with the real frames. The following example, using a server processing high-dynamic video data of "a person quickly turning around + gesturing," details the error calculation process before model training. The server obtains a packet loss video stream instance from the training dataset: this instance simulates a video conference recording (1000 frames, 30fps) with a 30% packet loss rate, including a complete scene of a person speaking from the front (frames 1-200) → quickly turning around to write on a whiteboard (frames 201-400, vigorous movement) → turning back to their original position to continue gesturing (frames 401-1000). 30% of the non-critical frames (P / B frames) transmitted via UDP are randomly lost (e.g., frames 250, 280, 320, etc.), while critical frames (I frames) are transmitted completely via TCP. The corresponding reference video stream instance is a packet loss-free version of the same video (original real frames, without any information loss), serving as the "gold standard" for compensation frame quality. The server inputs instances of lost video streams into an incompletely trained packet loss compensation model. The model, through an LSTM prediction network and a GAN generator network, generates compensation frames for lost non-critical frames (e.g., frame 250: a person turning around in a half-side-facing pose, holding a pen in their right hand to write), denoted as the "compensation frame generation result." Subsequently, the server activates a frame quality fluctuation detection unit to calculate the "frame detail fluctuation" of the compensation frame generation result from three dimensions: Spatial domain frame detail fluctuation: Gradient maps (edge ​​and texture details) of the compensation frame and reference frame are extracted using the Sobel operator, and the variance of the gradient values ​​is calculated (reflecting the stability of spatial details). For example, in frame 250, the gradient variance of the person's shoulder edge is 0.18 (relatively large fluctuation) due to motion blur, while the gradient variance of the background whiteboard edge is 0.05 (relatively small fluctuation). The overall spatial domain fluctuation is calculated using a region-weighted average of 0.12. Temporal domain frame detail fluctuation: The rate of change of motion vectors between the compensation frame and its preceding and following generated frames (frames 249 and 251) is calculated (motion vectors are extracted using optical flow). When the character turns, the waist motion vector abruptly changes from (-3 pixels / frame, left shift) in frame 249 to (+5 pixels / frame, right shift) in frame 250, with an absolute change rate of 8 and a temporal domain fluctuation of 0.15 (fluctuation caused by motion discontinuity). Spectral envelope feature fluctuation: Fourier transform is performed on the compensated frame and the reference frame to extract the power spectrum envelope of the high-frequency components (corresponding to detail textures), and the root mean square error (RMSE) of the envelope curve is calculated. In the compensated frame, the RMSE of the character's pen tip (high-frequency detail) is 0.21, and the spectral envelope fluctuation is 0.17 (fluctuation caused by loss of high-frequency details).The server calculates the "deviation" between the compensated frame generation result and the reference video stream instance: spatial domain detail deviation (the SSIM value of the compensated frame and the reference frame is 0.82, corresponding to a deviation of 0.18), temporal domain detail deviation (mean square error of motion vector is 2.3 pixels / frame), and spectral envelope feature deviation (high-frequency component energy is 15% lower than that of the reference frame). Combining frame detail fluctuations, the server calculates the training error according to a preset formula: First training stage error: using spatial domain fluctuation of 0.12 as the normalization coefficient, the spatial domain deviation of 0.18 is weighted (0.18 × 0.12 = 0.0216); temporal domain fluctuation of 0.15 is weighted with temporal domain deviation of 2.3 (2.3 × 0.15 = 0.345), the sum of which is the first error of 0.3666; spectral envelope fluctuation of 0.17 is weighted with spectral deviation of 0.15 (0.15 × 0.17 = 0.0255) as the second error; superimposed on visual distortion error (PSNR value 28dB, corresponding error 0.22) and state transition frequency error (attention gating switching frequency 1.2 times / frame, exceeding threshold 0.3 times / frame, error 0.09), the total first stage error is 0.3666 + 0.0255 + 0.22 + 0.09 = 0.6921. The second training stage error is as follows: the spatial domain volatility of 0.12 is directly weighted with the deviation of 0.18 (0.18 × 0.12 = 0.0216), and the temporal domain volatility of 0.15 is weighted with the deviation of 0.23 (0.23 × 0.15 = 0.0345), resulting in a third error of 0.0561; the spectral envelope volatility of 0.17 is weighted with the deviation of 0.15 by 0.0255, resulting in a fourth error; the same visual distortion error of 0.22 and the state transition frequency error of 0.09 are superimposed, resulting in a total second stage error of 0.0561 + 0.0255 + 0.22 + 0.09 = 0.3916. The server weights and averages the errors from the two stages (0.6 for the first stage and 0.4 for the second stage), resulting in a total training error of 0.6921×0.6 + 0.3916×0.4 ≈ 0.5719. The server then optimizes the model parameters (such as the attention gating weights of the LSTM and the convolution kernel parameters of the GAN generator) through the backpropagation algorithm, reducing the error to below 0.2 (corresponding to a PSNR ≥ 35dB between the compensation frame and the reference frame), thus completing this round of model training.

[0051] In this embodiment of the invention, the step of determining the training error value based on the compensation frame generation result of the lost video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation, and training the packet loss compensation model based on the training error value, can be implemented through the following example.

[0052] The training error value of the first training stage is determined based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instances, and the frame detail fluctuation. The training error value of the first training stage is used to regulate the stability of the frame detail fluctuation.

[0053] The training error value of the second training stage is determined based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation. The training error value of the second training stage is used to set the corresponding regional attention for the regional temporal component based on the regional credibility fluctuation of the regional temporal component when determining the compensation frame generation result of the packet loss video stream instance and the deviation between the reference video stream instance.

[0054] The packet loss compensation model is trained sequentially based on the training error values ​​of the first training stage and the second training stage.

[0055] In an embodiment of the invention, for example, in a video conferencing scenario, the server trains a model for a high-dynamic packet loss video stream instance (simulating a 30% packet loss rate, with the 250th frame being a non-critical frame lost, corresponding to the person's half-side-facing pen-holding posture) where a person turns to write on a whiteboard. At this time, the server has obtained key data through the frame quality fluctuation detection unit: the compensated frame generation result (the 250th frame generated by the model) deviates from the reference video stream instance (a real frame without packet loss) (spatial domain detail deviation manifests as blurred shoulder edges, temporal domain detail deviation manifests as discontinuous waist motion vectors, and spectral envelope feature deviation reflects the loss of high-frequency texture at the pen tip). Simultaneously, frame detail fluctuation is detected (spatial domain fluctuation is higher due to blurred shoulder edges, temporal domain fluctuation is larger due to abrupt changes in waist motion vectors, and spectral envelope fluctuation is significant due to the loss of pen tip texture). The following details how the server determines the training error and optimizes the model sequentially through the first and second training stages. The core objective of the first training stage is to suppress drastic fluctuations in the details of the compensated frame in the spatial, temporal, and spectral dimensions, ensuring the stability of the generated frames. The server dynamically adjusts the deviations using a "volatility normalization weighting" strategy. When the volatility of a certain dimension is high (the details are less stable), normalization reduces the impact of that dimension's deviation on the overall error, preventing the model from overemphasizing drastically fluctuating noise. Specifically, the server first handles spatial and temporal domain deviations: For the spatial domain, due to the blurring of shoulder edges causing high spatial volatility, the server normalizes the deviations using volatility (the higher the volatility, the lower the normalized deviation weight), weakening the interference of blurry edges on the overall error and ensuring the stability of spatial textures in the generated frames (such as clothing folds and whiteboard grid lines). For the temporal domain, due to the abrupt changes in waist motion vectors causing significant temporal volatility, the server also handles temporal domain deviations using volatility normalization, suppressing abrupt changes in motion vectors and making the transition of the character's turning motion smoother from frame 249 to frame 251 (e.g., in the change of waist rotation angle from 15° to 20° to the left, the angle deviation in intermediate frames is controlled within 3°). Subsequently, the server sums the normalized spatial and temporal domain deviations with their corresponding volatility to obtain a comprehensive error reflecting spatial and temporal stability. Next, the server addresses spectral envelope feature deviations: due to the loss of high-frequency texture at the pen tip, significant spectral envelope volatility occurs. The server normalizes the deviations based on this volatility, reducing high-frequency noise interference with details and ensuring that the sharpness of the pen tip in the generated frame (e.g., the edge clarity of the pen tip's contact point with the whiteboard) is consistent with the reference frame. Finally, the server superimposes visual distortion errors (differences between the overall visual effect of the compensation frame and the reference frame, such as skin tone distortion and uneven background brightness) and state transition frequency errors (instability in feature extraction caused by excessively frequent attention gating switching, such as frequent jumps in attention between the facial and hand regions) to obtain the total error of the first training stage.This error is used to guide the model to prioritize optimizing the stability of frame details. For example, adjusting the cell state update rate of the long short-term memory component in the LSTM prediction network reduces feature fluctuations caused by intense motion; adding a smoothing filter layer to the GAN generator network suppresses abrupt changes in high-frequency components. The core objective of the second training phase is to dynamically adjust the priority of attention to deviations in each region based on the region's credibility fluctuations (differences in volatility between different regions). For regions with high volatility (such as hands, pen tips, and other moving or detailed regions), the weight of their deviations in the error is increased to guide the model to prioritize optimizing the details of these regions; for regions with low volatility (such as background walls, desktops, and other static regions), the deviation weights are reduced to decrease redundant computation. Specifically, the server first addresses spatial and temporal domain biases: for the hand region (where temporal domain volatility is high due to variations in hand movement), the server directly uses the temporal domain volatility of this region as a weight to amplify the proportion of hand motion vector bias in the error, prompting the model to focus on learning the hand's movement trajectory from the table to the chest (such as wrist rotation angle and finger bending degree); for the shoulder edge (where spatial domain volatility is high), the server uses spatial domain volatility as a weight to amplify its spatial detail bias, driving the model to optimize the clarity of the shoulder contour (such as the transition between the shoulder and the edge of clothing folds). Subsequently, the server sums the weighted spatial and temporal domain biases to obtain a comprehensive error reflecting the priority of region attention. Next, the server addresses spectral envelope feature biases: for the pen tip region (where spectral envelope volatility is high and high-frequency texture is lost), the server uses spectral envelope volatility as a weight to amplify its spectral bias, guiding the model to enhance the high-frequency details of the pen tip (such as the metallic reflection of the pen tip and the texture of ink marks). Finally, the server also superimposes visual distortion error and state transition frequency error to obtain the total error of the second training stage. This error is used to guide the model in optimizing the allocation of regional attention, such as adjusting the regional attention confidence calculation logic of attention gating to increase the attention weight of high-fluctuation regions such as the hand and pen tip; and adding targeted regional feature fusion to the temporal prediction unit of the LSTM prediction network (such as using more refined local relation feature division for the pen tip region). The server adopts a strategy of "stabilizing fluctuations first, then optimizing attention" to train the model sequentially: First, based on the backpropagation of the error in the first training stage, the model parameters are adjusted to stabilize the frame detail fluctuations. Through 50 rounds of iterative training, the spatial domain fluctuation of the compensation frame is reduced from "high" to "stable" (the blurring of the shoulder edge is improved, and the texture consistency with the reference frame is improved), the temporal domain fluctuation is reduced from "large" to "smooth" (the jump amplitude of the waist motion vector is reduced, and the turning action is smooth), and the spectral envelope fluctuation is reduced from "significant" to "slight" (the high-frequency texture of the pen tip is restored, and the sharpness is close to that of the reference frame). At this time, the error of the first training stage is optimized to below the preset threshold, and the frame detail stability meets the standard.Subsequently, the server continued backpropagation based on the error from the second training stage, optimizing the allocation of regional attention. Through 30 rounds of iterative training, the model significantly improved the attention weight for high-fluctuation regions (hand, pen tip). The deviation of the hand motion vector decreased from "obvious" to "minor" (the hand movement trajectory was basically consistent with the reference frame), and the deviation of the high-frequency texture of the pen tip decreased from "significant" to "negligible" (the pen tip details were highly consistent with the reference frame). At this point, the error of the second training stage was optimized to below the preset threshold, and the accuracy of regional attention was achieved. Finally, after two training stages, the compensation frame generated by the packet loss compensation model under a 30% packet loss rate showed significantly improved visual consistency with the reference frame. The video stutter-free rate reached 95%, and the synchronization accuracy was controlled within 15ms, meeting the requirements of smoothness and real-time performance in video conferencing.

[0056] In this embodiment of the invention, the frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation, and spectral envelope feature fluctuation. The step of determining the training error value of the first training step based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation can be implemented through the following example.

[0057] Determine the compensation frame generation result of the packet loss video stream instance and the spatial domain detail deviation, temporal domain detail deviation, and spectral envelope feature deviation between the reference video stream instances;

[0058] The spatial domain detail deviation is weighted according to a Gaussian distribution function with the spatial domain frame detail fluctuation as the normalization coefficient, and the temporal domain detail deviation is weighted according to a Gaussian distribution function with the temporal domain frame detail fluctuation as the normalization coefficient. The first error is determined based on the sum of the spatial domain frame detail fluctuation, the temporal domain frame detail fluctuation, the weighted spatial domain detail deviation, and the weighted temporal domain detail deviation.

[0059] The spectral envelope feature deviation is weighted according to a Gaussian distribution function with the spectral envelope feature fluctuation as the normalization coefficient, and the second error is determined according to the sum of the spectral envelope feature fluctuation and the weighted spectral envelope feature deviation.

[0060] The training error value of the first training step is obtained by weighting and summing the first error and the second error.

[0061] In an embodiment of the invention, exemplarily, in a video conferencing scenario, the server performs error calculations for the first training stage on a high-dynamic packet loss video stream instance (simulating a 30% packet loss rate, with the 250th frame being a non-critical frame lost, corresponding to the posture of a person half-turning to hold a pen and prepare to write) where "a person turns to write on a whiteboard." At this time, the server has obtained key data through the frame quality fluctuation detection unit: the detail deviation between the compensation frame generation result (the 250th frame generated by the model) and the reference video stream instance (a real frame without packet loss), as well as the frame detail fluctuation in various dimensions (the spatial domain fluctuation is higher due to the blurring of the shoulder edges, the temporal domain fluctuation is larger due to the jump in the waist motion vector, and the spectral envelope fluctuation is significant due to the loss of high-frequency texture at the pen tip). The server determines the training error value of the first training stage according to the following steps to regulate the stability of the frame detail fluctuation. The server first clarifies the detail deviation between the compensation frame generation result and the reference video stream instance through multi-dimensional comparison: Spatial domain detail deviation: manifested as the difference between the blurring of the person's shoulder edges in the compensation frame and the clear edges in the reference frame. In the reference frame, the edges of the clothing folds (cotton and linen texture) on the left shoulder of the figure due to turning are clearly visible. In the compensated frame, a 1-2 pixel blurry halo appears at the edge of this area, weakening the sense of depth in the folds. Simultaneously, the grid lines (1 pixel wide black solid lines) of the white background board show local pixel breaks in the compensated frame (e.g., the break in the grid line at row 3, column 5), creating a visual discrepancy with the continuous lines in the reference frame. Temporal domain detail deviation: This manifests as discontinuity in the motion vectors between the compensated frame and the preceding and following frames (frames 249 and 251). In the reference frame, the rotation angle of the figure's waist due to the turning motion smoothly transitions from a 15° leftward tilt in frame 249 to a 20° rightward tilt in frame 251, with the angle in frame 250 being a 5° leftward tilt. However, in the compensated frame, the waist angle jumps abruptly to a 10° rightward tilt, showing a clear disconnect from the motion trend of the preceding and following frames, resulting in a "stuttering" feeling in the turning motion. Spectral envelope feature deviation: This is reflected in the lack of energy in high-frequency details. In the reference frame, the pen tip (metallic material) held by the figure exhibits subtle reflective textures (rich in high-frequency components), with high-frequency (>10kHz) energy accounting for 22% after Fourier transform. In the compensation frame, the reflective texture of the pen tip is blurred, and the high-frequency energy accounts for only 12%, a 10% energy difference compared to the reference frame, resulting in insufficient "sharpness" of the pen tip. The server uses a Gaussian distribution function to weight spatial and temporal domain biases. The core principle is to use the "frame detail fluctuation" of the corresponding dimension as a normalization coefficient to dynamically adjust the impact of bias on the error. When the fluctuation of a certain dimension is higher (the more unstable the details), the Gaussian distribution function will suppress the weight of the bias in that dimension through normalization, avoiding the model from over-focusing on drastically fluctuating "noise details" and prioritizing the stabilization of overall features. Due to the blurry shoulder edges and broken grid lines, the spatial domain frame detail fluctuation is high (poor detail stability).The server uses spatial domain volatility as a normalization coefficient and assigns weights to spatial domain detail deviations using a Gaussian distribution function: the higher the volatility, the stronger the "suppression effect" of the Gaussian function, thus weakening the interference of blurred edges and grid line breakpoints on the overall error. For example, for shoulder wrinkle blur deviation, the "effective weight" of the deviation is reduced after normalization (avoiding the model from over-adjusting the overall texture due to local blur), ensuring that the spatial structure of the generated frame (such as the outline of the character's torso and the overall layout of the whiteboard) remains consistent with the reference frame; for grid line break deviation, the deviation weight is also reduced after normalization, so that the model prioritizes the "continuity" of the grid lines rather than individual breakpoints, avoiding distortion of the entire line due to repairing breakpoints. Due to the jump in waist motion vectors, the temporal domain frame detail volatility is relatively large (poor motion continuity). The server uses temporal domain volatility as a normalization coefficient and assigns weights to temporal domain detail deviations using a Gaussian distribution function: the higher the volatility, the stronger the "smoothing effect" of the Gaussian function on the deviation, suppressing abrupt changes in motion vectors. For example, to address waist angle jump deviations, the "effective weight" of the normalized deviation is reduced (to avoid overcompensation by the model to correct jumps, leading to new motion distortions), making the transition of the character's turning action from frame 249 to frame 251 more natural (e.g., adjusting the waist rotation angular velocity from "abrupt 30° / frame" to "gradual 15° / frame"), ensuring temporal smoothness. The server directly accumulates the weighted spatial domain detail deviation, the weighted temporal domain detail deviation, and the corresponding dimension's frame detail fluctuation (spatial domain fluctuation, temporal domain fluctuation) to obtain the "first error" reflecting spatial and temporal stability. This error comprehensively reflects the overall deviation of the compensation frame in terms of spatial texture continuity (such as wrinkles and grid lines) and temporal motion smoothness (such as turning angle). For example, the spatial domain volatility (basic volatility caused by edge blurring) and the weighted spatial deviation (effective interference from blurred edges) are superimposed, along with the temporal domain volatility (basic volatility caused by motion jumps) and the weighted temporal deviation (effective interference from jumps), together constituting the first error. Its value decreases as the stability of spatial and temporal details improves. The server uses the same logic as in the spatial and temporal domains to process the spectral envelope feature deviation: due to the loss of high-frequency texture at the pen tip, the spectral envelope frame detail volatility is significant (poor high-frequency stability). The server uses the spectral envelope volatility as a normalization coefficient and assigns weights to the spectral envelope feature deviation through a Gaussian distribution function. The higher the volatility, the stronger the "filtering effect" of the Gaussian function on the deviation, avoiding interference from high-frequency noise (such as random noise from pen tip reflections) on details. For example, to address the bias of missing high-frequency energy at the pen tip, the "effective weight" of the bias is reduced after normalization (to avoid introducing unnecessary noise into the model to increase high-frequency energy), ensuring that the high-frequency details of the generated frame (such as pen tip reflection and the sharpness of the whiteboard writing edges) are consistent with the reference frame (such as the pen tip reflection texture being corrected from "blurry light spots" to "clear striped reflections", and the proportion of high-frequency energy increasing from 12% to 18%).Subsequently, the server sums the weighted spectral envelope feature deviation with the spectral envelope frame detail fluctuation (basic fluctuation caused by high-frequency loss) to obtain the "second error" reflecting the stability of high-frequency details. Based on the priority of the video content (spatial and temporal stability take precedence over high-frequency details, as the smoothness of human movements is more important than pen glare in a meeting scene), the server assigns a higher weight (e.g., 60%) to the first error (spatial + temporal stability) and a lower weight (e.g., 40%) to the second error (spectral envelope stability). The weighted sum of these two errors yields the "training error value of the first training stage." For example, the first error, due to spatial edge blurring and temporal motion jumps, is considered a "higher error," while the second error, due to high-frequency detail loss, is considered a "medium error." After weighted summation, the total error directly reflects the current compensation frame's deficiency in "overall stability." The server backpropagates this error to the packet loss compensation model, adjusts the cell state update threshold of the long short-term memory component in the LSTM prediction network (reducing the state update frequency for highly volatile regions), and adds a smoothing filter kernel (3×3 mean filter) to the convolutional layers of the GAN generator network to suppress excessive fluctuations at spatial domain edges and abrupt changes in temporal motion. After 20 rounds of iterative training, the spatial domain edge blurring in frame 250 is reduced (shoulder wrinkles are clearly visible), and the temporal motion vector is continuous (smooth transition of waist rotation angle). The error in the first training stage is reduced to below the preset threshold, and the stability of frame detail fluctuations meets the standard.

[0062] In this embodiment of the invention, the step of weighted summation of the first error and the second error to obtain the training error value of the first training stage can be implemented through the following example.

[0063] The visual distortion error is determined based on the compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and the state transition frequency error is determined based on the deviation between the state transition frequency of the attention gating and the preset frequency threshold.

[0064] The first error and the second error are weighted and summed to obtain the first region credibility fluctuation error. The first region credibility fluctuation error, the visual distortion error, and the state transition frequency error are weighted and summed to obtain the training error value of the first training step.

[0065] In an embodiment of the present invention, for example, in a video conferencing scenario, the server calculates the training error value of the first training stage for a video stream instance with packet loss (the 250th frame is a non-key frame and the compensation frame corresponds to the person's half-side-facing pen-holding posture) where "a person turns to write on a whiteboard". This requires considering the fluctuation of regional credibility, visual distortion, and the stability of attention gating state transition. The specific process is as follows: The server first determines the visual distortion error through subjective visual quality assessment and objective parameter detection: the compensation frame generation result differs from the reference video stream instance (real frames without packet loss) in overall visual effect. In the reference frame, the person's facial skin color is a natural yellow skin color (RGB mean approximately 230, 190, 160). In the compensation frame, due to the color mapping deviation of the GAN generator, the skin color is reddish (RGB mean 240, 180, 150), and there is a slight color shift in the cheek area. The whiteboard background has uniform brightness in the reference frame (grayscale value 240±5), but the upper left corner of the compensation frame has lower brightness due to light estimation deviation (grayscale value 220±8), creating a difference in brightness with the lower right corner. Considering both color cast and brightness unevenness, the visual distortion error is classified as "moderate distortion" (corresponding to a quantization value of 0.25, where 0 represents no distortion and 1 represents severe distortion). Subsequently, the server detects state transition frequency error: attention gating needs to dynamically allocate attention across different regions (face, hands, background), with a preset frequency threshold of "1 time / frame" (to avoid frequent switching leading to unstable feature extraction). During compensation frame generation, the attention gating experiences two state transitions within one frame due to shoulder edge blurring and waist movement abrupt changes (switching from "face attention" to "hand attention," then back to "background attention"). This exceeds the threshold by one time, resulting in a state transition frequency deviation of "1 time / frame" (corresponding to a quantization value of 0.15, where 0 represents no deviation and 1 represents severe deviation). The server weights and accumulates the first error (spatial + temporal stability error) and the second error (spectral envelope stability error) according to scene priority: because the fluidity of human movements (spatial + temporal stability) is more important than the high-frequency texture of the pen tip (spectral envelope) in a meeting scene, the first error has a weight of 60%, and the second error has a weight of 40%. The first error, due to the blurring of the shoulder edge and the abrupt changes in temporal motion, manifests as a "higher fluctuation error" (quantization value 0.6); the second error, due to the loss of high-frequency texture at the pen tip, manifests as a "medium fluctuation error" (quantization value 0.4). After weighted summation: the first region's credibility fluctuation error = 0.6 × 60% + 0.4 × 40% = 0.36 + 0.16 = 0.52 (reflecting the overall credibility defect of regional detail fluctuations).The server prioritizes video quality (regional fluctuation stability > visual distortion > state transition frequency), assigning a 70% weight to the first region's reliability fluctuation error (0.52), a 20% weight to the visual distortion error (0.25), and a 10% weight to the state transition frequency error (0.15). The weighted sum of these three factors results in the training error value for the first training phase: 0.52 × 70% + 0.25 × 20% + 0.15 × 10% = 0.364 + 0.05 + 0.015 = 0.429 (quantization value < 0.5, initially meeting the fluctuation stability requirements, but further optimization is needed). This error value guides the server's backpropagation adjustments to the model: for visual distortion, optimizing the color mapping layer parameters of the GAN generator (correcting the RGB mean of skin color to 230, 190, 160); for state transition frequency, reducing the region switching sensitivity of the attention gating (relaxing the switching threshold from 1 time / frame to 1.2 times / frame), ultimately significantly improving the overall stability and visual consistency of the compensated frames.

[0066] In this embodiment of the invention, the frame detail fluctuation includes spatial domain frame detail fluctuation, temporal domain frame detail fluctuation, and spectral envelope feature fluctuation. The step of determining the training error value of the second training step based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation can be implemented through the following example.

[0067] The spatial domain detail deviation is weighted according to the spatial domain frame detail fluctuation, the temporal domain detail deviation is weighted according to the temporal domain frame detail fluctuation, and the third error is determined according to the sum of the weighted spatial domain detail deviation and the weighted temporal domain detail deviation.

[0068] The spectral envelope feature deviation is weighted according to the spectral envelope feature fluctuation, and the weighted spectral envelope feature deviation is determined as the fourth error;

[0069] The training error value of the second training stage is obtained by weighting and summing the third error and the fourth error.

[0070] In an embodiment of the invention, for example, in a video conferencing scenario, the server calculates the training error value of the second training stage for a packet-loss video stream instance where "a person turns to write on a whiteboard" (the 250th frame is a non-key frame and the compensation frame corresponds to the person's half-side-facing pen-holding posture). The core objective is to dynamically adjust the deviation weights based on the fluctuation of regional credibility (prioritizing optimization of high-fluctuation regions), focusing on improving the detail quality of high-information regions such as the hand and pen tip. The specific process is as follows: The server focuses on the detail deviations in high-fluctuation regions and directly assigns weights to the deviations using "frame detail fluctuation." The higher the fluctuation (the more unstable the regional details), the greater the deviation weight, ensuring that the model prioritizes optimization of these key regions. Spatial domain detail deviation weighting: In the compensation frame, the area where the person holds the pen in their right hand experiences spatial domain detail deviation due to motion blur (blurred hand contour edges and lost finger joint texture). In the reference frame, the finger joint protrusions (such as the second joint of the index finger) in this area are clearly visible. In the compensation frame, a 2-3 pixel blur band appears at the edge of this joint, and the deviation is expressed as "relatively large deviation" (quantization value 0.3). Due to the intense hand movements, the spatial domain frame detail fluctuation is "high" (quantization value 0.6, 0 for stable, 1 for extremely unstable). After weighting, the spatial domain deviation = relatively large deviation × high fluctuation = 0.3 × 0.6 = 0.18 (weight increased, requiring key optimization). Temporal domain detail deviation weighting: The hand motion vector in the compensation frame jumps between the previous and next frames (frames 249 and 251), resulting in significant temporal domain detail deviation. In the reference frame, the hand moves from the "initial desktop position" (coordinates (120, 180)) in frame 249 to the "chest writing position" (coordinates (160, 150)) in frame 251, and the motion vector should increase smoothly (moving 20 pixels per frame). In the compensation frame, the hand position abruptly changes to (170, 140), with a difference of 50 pixels from the motion vector in frame 249, exhibiting a "significant deviation" (quantization value 0.4). Due to the intense hand movements, the temporal domain frame detail fluctuation is "large" (quantization value 0.5). After weighting, the temporal domain deviation = significant deviation × large fluctuation = 0.4 × 0.5 = 0.2 (weight increased, motion continuity needs to be corrected first). The server adds the weighted spatial domain deviation (0.18) to the temporal domain deviation (0.2) to obtain the "third error" = 0.18 + 0.2 = 0.38 (reflecting the comprehensive optimization needs of spatial details and temporal motion in the hand area; the higher the error value, the higher the optimization priority). The server directly weights the spectral envelope feature deviation of the high-frequency detail area (pen tip) with "spectral envelope feature fluctuation" to strengthen the optimization of high-frequency textures. In the compensation frame, the pen tip (metal material) has a spectral envelope feature deviation due to the loss of high-frequency texture: the pen tip reflection texture (0.5 pixel width and height light bar) in the reference frame is clear, and the high-frequency band (>10kHz) energy accounts for 22% after Fourier transform; the pen tip reflection in the compensation frame is blurred into "diffuse light spots", and the high-frequency band energy accounts for only 12%, and the deviation is "moderate deviation" (quantization value 0.35).Since the pen tip is the core area of ​​the writing action, its spectral envelope feature fluctuation is "significant" (quantization value 0.7). After weighting, the spectral envelope feature deviation = moderate deviation × significant fluctuation = 0.35 × 0.7 = 0.245, that is, the "fourth error" = 0.245 (reflecting the optimization needs of high-frequency details of the pen tip). Based on the priority of "hand movements (spatial + temporal)" and "pen tip details (spectral)" in the meeting scenario (hand movement fluency directly affects information transmission and has a higher weight), the server assigns 70% weight to the third error and 30% weight to the fourth error, and adds the two weights together: the training error value of the second training stage = third error × 70% + fourth error × 30% = 0.38 × 0.7 + 0.245 × 0.3 = 0.266 + 0.0735 = 0.3395 (error value > 0.3, indicating that the hand and pen tip areas need to be optimized). The error guides the server to backpropagate and adjust the model: For the third error (spatial and temporal bias of the hand), the attention weight of the hand region in the LSTM prediction network is increased (from 0.38 to 0.5), enhancing the prediction accuracy of hand motion vectors; for the fourth error (pen tip spectral bias), the learning rate of the high-frequency convolutional layer in the GAN generator network is increased (from 0.001 to 0.003), strengthening the high-frequency components of pen tip reflective texture. After 25 training epochs, the hand knuckle texture in the compensation frame is restored (the blur band is reduced to 0.5 pixels), the motion vector jump is reduced to within 15 pixels, the proportion of high-frequency energy in the pen tip is increased to 19%, the error of the second training stage is optimized to 0.28 (<0.3), and the regional attention allocation and detail quality are significantly improved.

[0071] In this embodiment of the invention, the step of weighted summation of the third error and the fourth error to obtain the training error value of the second training stage can be implemented through the following example.

[0072] The visual distortion error is determined based on the compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and the state transition frequency error is determined based on the deviation between the state transition frequency of the attention gating and the preset frequency threshold.

[0073] The third error and the fourth error are weighted and summed to obtain the second region credibility fluctuation error. The second region credibility fluctuation error, the visual distortion error, and the state transition frequency error are weighted and summed to obtain the training error value of the second training stage.

[0074] In an embodiment of the invention, for example, in a video conferencing scenario, the server calculates the training error value of the second training stage for a lost video stream instance of "a person turning to write on a whiteboard" (the 250th frame is a non-key frame and the compensation frame corresponds to the person's half-side-facing pen-holding posture). The core is to guide the model to prioritize optimizing the detail quality of high-information areas such as the hand and pen tip through a comprehensive evaluation of regional credibility fluctuations, visual distortion, and state transition frequency. The specific process is as follows: The server first evaluates the visual distortion error between the compensation frame generation result and the reference video stream instance: After optimization in the first training stage, the overall visual distortion is reduced, but residual distortion still exists in high-fluctuation areas. Due to the blurring of the knuckles (spatial domain deviation), the hand contour of the person holding the pen in the right hand has a slight difference from the reference frame (the knuckles are clearly raised in the reference frame, but there is still a 1-pixel blurry band in the compensation frame); the pen tip reflective texture (spectral envelope deviation) is improved from "diffuse light spots" to "striped reflections", but the length is 1 pixel shorter than that of the reference frame. After considering residual distortion in the overall region, the visual distortion error was reduced to "slight distortion" (corresponding to a quantization value of 0.2, where 0 represents no distortion and 1 represents severe distortion). Subsequently, the server detected the state transition frequency error: after the first stage of optimization, the state transition frequency of the attention gating decreased from 2 times / frame to 1.5 times / frame (still exceeding the preset threshold of 1 time / frame), with a deviation of 0.5 times / frame (corresponding to a quantization value of 0.1, where 0 represents no deviation and 1 represents severe deviation). At this point, the attention gating mainly switched between "hand attention" and "pen tip attention" (because these two regions have high volatility, the weights need to be dynamically adjusted). Although the switching frequency was reduced, it still needs further stabilization to avoid fluctuations in feature extraction. The server weighted and accumulated the third error (hand spatial + temporal stability error, 0.38) and the fourth error (pen tip spectral envelope stability error, 0.245) according to region priority: because hand movements (conveying writing intention) are more critical than pen tip reflection (auxiliary details) in the meeting scenario, the third error was given a 70% weight, and the fourth error was given a 30% weight. The second region's reliability fluctuation error = third error × 70% + fourth error × 30% = 0.38 × 0.7 + 0.245 × 0.3 = 0.266 + 0.0735 = 0.3395 (approximately 0.34, reflecting the urgency of optimizing details in high-fluctuation regions; the higher the error value, the more priority should be given to adjusting the model's focus on that region). Based on the priority of "regional detail quality > overall visual effect > state transition stability," the server assigns a 70% weight to the second region's reliability fluctuation error (0.34), a 20% weight to the visual distortion error (0.2), and a 10% weight to the state transition frequency error (0.1). The weighted sum of these three is: the training error value of the second training stage = 0.34 × 70% + 0.2 × 20% + 0.1 × 10% = 0.238 + 0.04 + 0.01 = 0.288 (approximately 0.29, error value < 0.3, close to the optimization target).The error guides the server to adjust the model through backpropagation: For the second region confidence fluctuation error (0.34), the weights of the temporal prediction units in the hand region of the LSTM prediction network are increased (from 0.45 to 0.55) to strengthen the continuous learning of motion vectors; for the visual distortion error (0.2), the color mapping layer parameters of the GAN generator are fine-tuned (correcting the RGB channel ratio of the pen tip reflection); for the state transition frequency error (0.1), the region switching sensitivity threshold of the attention gating is reduced (from 1 time / frame to 1.2 times / frame). After 20 rounds of training, the blurry band of the hand knuckles in the compensation frame is eliminated, the pen tip reflection length is consistent with the reference frame, the error of the second training stage is optimized to 0.25, and the regional detail quality and attention allocation meet the standards.

[0075] In this embodiment of the invention, the process of performing region enhancement mapping on the first temporal feature to obtain the second temporal feature can be implemented through the following example.

[0076] The first temporal feature is divided based on the spatial dimension to obtain the first local relation feature and the second local relation feature;

[0077] The first local relation features are standardized, and the standardized first local relation features are spatially filtered in the local region dimension based on the time sequence dimension to obtain the first feature map.

[0078] Determine the first multiplication result of the first feature map and the second local relation feature, and obtain the second temporal feature based on the sum of the first multiplication result and the second local relation feature.

[0079] In an embodiment of the present invention, for example, in a video conferencing scenario, the server, for the compensation process of the lost 250th frame, has obtained the first temporal feature (a 64×64×256 feature map, including motion-related features of the person's face, hands, and background, focusing on capturing the spatial relationship between the motion vector of the person's right hand holding a pen and the shoulder contour when turning around) through inter-frame motion mapping. The server needs to perform region enhancement mapping on this feature to highlight the details of high-information areas such as the hands and face, and obtain the second temporal feature. The specific process is as follows: The server divides the first temporal feature into regions based on the spatial dimension, focusing on the differential enhancement of high-information areas and low-information areas: First local relationship feature: corresponding to the area of ​​the person's right hand holding a pen (high-information area, including key details such as finger joint movement and pen tip trajectory) and the facial area (including facial expression changes during head-raising movements). The server extracts feature sub-maps of these two regions from the first temporal feature using a sliding window (16×16 pixels), with dimensions of 16×16×256, including hand motion vectors (2.5 pixels / frame horizontally to the right, 3 pixels / frame vertically upward) and facial vertical displacement (head tilt rate 1.2 pixels / temporal stage). The second local relational feature: corresponding to the background region (low-information areas, static features such as conference room tables and chairs, whiteboard edges, etc.), extracts feature sub-maps of the remaining regions, with dimensions of 48×48×256, including basic background textures (such as the direction of the whiteboard grid lines, desktop wood grain). The server performs standardization processing on the first local relational feature (hand + face sub-maps) (LayerNorm, normalization along the channel dimension to eliminate dimensional differences between different feature channels, such as unifying the numerical range of hand motion vectors and facial texture features). Subsequently, spatial filtering is performed on the local region dimension based on the temporal dimension: a 3×3 convolution kernel (with weights optimized for hand joint texture to enhance edge response) is used to filter the standardized sub-image, highlighting the raised contours of the finger joints (such as the gradient change of the second joint of the index finger) and the trajectory of the pen tip (a straight line vector from the desktop to the chest), outputting the "first feature map" (16×16×256), which contains refined motion and texture features of the hand and face regions (such as contour deformation caused by hand muscle contraction and the height change of the brow bone when the face is raised). The server fuses the features of high-information regions and low-information regions through a "multiplication-addition" operation, enhancing the details of key regions while preserving background information: the first multiplication result: calculates the element-wise multiplication of the first feature map (hand + refined facial features) and the second local relation features (background features), highlighting the correlation between high-information regions and the background (such as the local light and shadow changes in the background caused by hand movement, and the relative positional relationship between the pen tip and the whiteboard grid lines when the pen tip moves). Addition and fusion: The result of the first multiplication is directly added to the second local relational feature. While preserving the basic texture of the background (such as the continuity of the whiteboard grid lines and the direction of the wood grain on the desktop), the refined features of the hands and face are injected into the overall feature map.Finally, the server outputs the "second temporal feature" (64×64×256). In this feature map, the clarity of the knuckle outline of the right hand holding the pen is improved by 30% (the blur band is reduced from 2 pixels to 0.5 pixels), and the continuity error of the pen tip motion vector is reduced to 1.5 pixels / frame (meeting the compensation requirements for hand movement details in the meeting scene). At the same time, the static features of the background area (such as the straightness of the whiteboard edge) remain stable, laying the foundation for subsequent regional feature fusion.

[0080] In this embodiment of the invention, the step of dividing the first temporal feature based on spatial dimension to obtain the first local relation feature and the second local relation feature can be implemented through the following example.

[0081] Based on the temporal dimension, the first temporal feature is spatially filtered in the spatial dimension to obtain the second feature map;

[0082] The second feature map is standardized, and the standardized second feature map is divided based on spatial dimensions to obtain the first local relation feature and the second local relation feature.

[0083] In an embodiment of the invention, exemplarily in a video conferencing scenario, the server, for the compensation process of the lost non-critical frame 250, has obtained the first temporal feature (a 64×64×256 feature map, including the hand motion vector when a person turns around and holds a pen, the facial head-raising action, and the associated features of the background static texture) through inter-frame motion mapping. To highlight the details of high-information areas (hands, face), the server needs to perform spatial dimension division on the first temporal feature to obtain the first and second local relationship features. The specific process is as follows: The server first performs spatial filtering on the first temporal feature based on the temporal dimension to enhance regional correlation and suppress noise. A 3×3 convolutional kernel (stride 1, padding 1, weights optimized through pre-training, focusing on enhancing the continuity of motion regions) was used to filter the first temporal features: For the area where the person is holding a pen with their right hand, the continuity of the hand motion vector (from the desktop position in frame 249 to the chest position in frame 251) was improved after filtering, and the horizontal displacement error was reduced from 3 pixels / frame to 1.5 pixels / frame; For the facial area, the vertical displacement (brow bone height change) of the head-raising action was smoother after filtering, avoiding facial contour distortion caused by temporal jumps; For the background area, the continuity of the whiteboard grid lines was enhanced after filtering, local breakpoints (such as the grid line in row 3, column 5) were repaired, and the noise (random pixel fluctuations) of the desktop wood grain texture was reduced by 30%. The final output is a "second feature map" (64×64×256), which significantly improves the correlation between features within the region while preserving the original spatial structure (such as a clearer linkage between hand movements and facial expressions). The server performs standardization on the second feature map (LayerNorm, which calculates the mean and variance along the channel dimension and normalizes the feature values ​​of each channel to a distribution with a mean of 0 and a variance of 1) to eliminate the dimensional differences of features in different regions. For example, after standardization, the hand motion vector (numerical range -5 to 5 pixels) and the background texture features (numerical range 0 to 255 grayscale values) are unified into dimensionless values ​​of -1 to 1, ensuring that the feature weights are comparable when segmenting in the future. Subsequently, the server divides the standardized second feature map into regions based on spatial dimensions: using a 16×16 pixel sliding window, feature sub-images of high-information regions are extracted as "first local relation features", specifically the region of the person holding a pen in their right hand (including finger joint movements and pen tip trajectory, corresponding to feature map coordinates (20~35, 30~45)) and the facial region (including changes in the brow bone and chin contours when looking up, corresponding to coordinates (10~25, 15~30)). The sub-image dimensions are 16×16×256, including refined features such as contour deformation caused by hand muscle contraction and subtle movements of facial expression muscles; the remaining region (background region, including static features such as conference room tables and chairs, whiteboard edges, etc., the part in coordinates (0~63, 0~63) excluding the above regions) is used as "second local relation features", with a sub-image dimension of 48×48×256, including the basic texture of the background (such as the direction of the whiteboard grid lines and the periodic distribution of the wood grain on the table).Through the above division, the server achieved feature separation between high-information-content regions and low-information-content regions, laying the foundation for subsequent targeted enhancement of details in high-information-content regions.

[0084] In this embodiment of the invention, the step of obtaining the second temporal feature based on the sum of the first multiplication result and the second local relational feature can be implemented through the following examples.

[0085] Based on the temporal dimension, the sum of the first multiplication result and the second local relational feature is spatially filtered in the spatial dimension to obtain the second temporal feature.

[0086] In an embodiment of the invention, exemplarily, during the compensation process of the 250th frame of a video conference, the server has calculated the sum of the first multiplication result (the correlation features between refined hand / face features and background features) and the second local relational feature (background basic texture). This sum includes the fusion information of the hand knuckle contours, pen tip trajectory, and background grid lines. Based on the temporal dimension, the server performs a 3×3 convolutional filtering (stride 1, padding 1, convolution kernel weights optimize the smoothness of the transition region) on the sum in the spatial dimension, suppressing abrupt transitions between the hand edges and the background (e.g., reducing the blurry area of ​​the right-hand pen-holding region from 2 pixels to 0.5 pixels), while preserving the orthogonality between the straight line vector of the pen tip trajectory and the background grid lines. The final result is the second temporal feature (64×64×256), significantly enhancing the details of high-information areas (hands, face).

[0087] In this embodiment of the invention, the step of performing regional feature fusion on the second temporal feature to obtain the output of the temporal prediction unit located first can be implemented through the following example.

[0088] The second temporal feature is standardized, and the standardized second temporal feature is then subjected to parallel spatial filtering to obtain the third and fourth feature maps after regional feature fusion.

[0089] The fourth feature map is sigmoid scaled to obtain feature selection weights;

[0090] The third feature map is weighted based on the feature selection weights to obtain the output of the time-series prediction unit that is ranked first.

[0091] In an embodiment of the invention, for example, in the compensation process of the 250th frame of a video conference, the server has obtained the second temporal feature (a 64×64×256 feature map, including fine movements of the hand knuckles, facial head-raising contours, and background whiteboard texture, with natural transitions between high-information areas and the background). It needs to dynamically adjust the weights of different regions through regional feature fusion to output the feature results of the first temporal prediction unit, providing input for the next unit of the subsequent LSTM prediction network. The specific process is as follows: The server first performs standardization processing on the second temporal feature (BatchNorm, normalization along the batch dimension, adjusting the feature values ​​to a distribution with a mean of 0 and a variance of 1), eliminating the dimensional differences between different regional features. For example, the gradient changes of the hand knuckles (numerical range -2~2) and the texture intensity of the background whiteboard grid lines (numerical range 0~1) are standardized to a dimensionless value of -1~1, ensuring that the feature weights are comparable during subsequent filtering. Subsequently, the server performs parallel spatial filtering on the standardized second temporal features, extracting global semantic features and region weight features through two convolutional layers: The third feature map: A 3×3 convolutional kernel (stride 1, padding 1, weights optimized for the overall posture of the person, such as torso rotation angle and spatial relationship between hands and face) is used for filtering, outputting a global semantic feature map (64×64×128), containing the overall outline of the person holding a pen in a semi-sideways position (e.g., waist rotation angle 30°, relative distance between right hand and face 20 pixels), and the hierarchical relationship between the background and the person (depth difference between the foreground and background whiteboard). The fourth feature map: A 1×1 convolutional kernel (no spatial expansion, focusing on channel dimension weight allocation) is used for filtering, outputting a region weight feature map (64×64×128), used for subsequent generation of feature selection weights (highlighting the focus on high-information areas). The server applies a Sigmoid function to scale the fourth feature map, compressing the feature values ​​to the range of 0~1, generating "feature selection weights" (64×64×128). The weights are dynamically allocated based on the amount of information in each region: the hand region (knuckles, pen tip) is key due to its motion vectors (affecting the transmission of writing intent), with a weight of 0.8~0.85 (high attention); the facial region (head tilt, brow bone contour) is important due to its facial expression information (aiding communication and understanding), with a weight of 0.7~0.75 (medium to high attention); the background region (whiteboard grid lines, desktop wood grain) is static and has low information content, with a weight of 0.15~0.2 (low attention). The weight matrix clearly distinguishes the region priorities, ensuring that high-information-content region features are prioritized during subsequent weighting.The server performs element-wise multiplication of the feature selection weights with the third feature map (global semantic features), giving stronger representation to features in high-weight regions: Hand region features (knuckle motion vectors, pen tip trajectory), after being weighted 0.8, show a 40% improvement in outline sharpness (knuckle protrusion edges are sharpened from 1 pixel to 0.5 pixels); Facial region features (chin angle when looking up, brow bone height), after being weighted 0.7, show a 30% enhancement in the feature response of facial details (such as a slight upturn of the mouth); Background region features (whiteboard grid line continuity), after being weighted 0.2, retain only basic textures, avoiding redundant information interference. Finally, the server obtains the "output of the first temporal prediction unit" (64×64×128 feature map), which highlights key features in high-information regions such as the hand and face, while suppressing background noise. This provides accurate local-global fusion features for the next temporal prediction unit (the second unit) of the LSTM prediction network, supporting the generation of subsequent prediction feature sequences.

[0092] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned deep learning-based video packet loss synchronization compensation method. Figure 2 As shown, Figure 2 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0093] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A video packet loss synchronization compensation method based on deep learning, characterized in that, include: The video stream to be compensated is acquired, and frame features are extracted from the video stream to obtain the original video frame features. The original video frame features are then loaded into the packet loss compensation model. The packet loss compensation model includes an LSTM prediction network and a GAN generator network set in sequence. Both the LSTM prediction network and the GAN generator network contain multiple sequentially connected temporal prediction units. Each temporal prediction unit includes a long short-term memory component. The long short-term memory component is equipped with an attention gating mechanism. The attention gating mechanism is used to guide the adjustment of the internal temporal state of the long short-term memory component or to maintain the internal temporal state constant. Based on the first temporal prediction unit in the LSTM prediction network, inter-frame motion mapping is performed on the original video frame features to obtain a first temporal feature, region enhancement mapping is performed on the first temporal feature to obtain a second temporal feature, and region feature fusion is performed on the second temporal feature to obtain the output of the first temporal prediction unit. The output of the first temporal prediction unit is then loaded into the next temporal prediction unit until the prediction feature sequence of the LSTM prediction network is obtained. Based on the GAN generator network, the predicted feature sequence of the LSTM prediction network is generated and decoded to obtain the target replacement frame features; Based on the target replacement frame features, frames are synchronously generated from the video stream to be compensated to obtain synchronous compensation frames.

2. The method according to claim 1, characterized in that, The step of performing inter-frame motion mapping on the original video frame features to obtain the first temporal features includes: The original video frame features are standardized, and the standardized original video frame features are then subjected to inter-frame motion mapping through the long short-term memory component. The output of the long short-term memory component is integrated along a time-series dimension to obtain the first time-series feature.

3. The method according to claim 2, characterized in that, The packet loss compensation model is configured to perform cyclic processing through multiple time stages. The step of performing inter-frame motion mapping on the standardized original video frame features using the Long Short-Term Memory component includes: For any current stage among the multiple time-series stages, obtain the initial region attention confidence and initial region attention confidence fluctuation of the long short-term memory component in the previous time-series stage of the current stage, and determine the target region attention confidence of the long short-term memory component in the current stage based on the initial region attention confidence and the initial region attention confidence fluctuation. The attention confidence of the target region is quantified to obtain the attention weight corresponding to the attention gating in the current stage. The internal temporal state of the Long Short-Term Memory (LSTM) component in the current stage is determined based on the attention weights, and inter-frame motion mapping is performed on the standardized original video frame features through the LSM component based on the internal temporal state.

4. The method according to claim 1, characterized in that, The packet loss compensation model further includes a frame quality fluctuation detection unit. Before loading the original video frame features into the packet loss compensation model, the method further includes: Obtain a lost video stream instance and a corresponding reference video stream instance, and call the packet loss compensation model to generate packet loss compensation for the lost video stream instance; The frame detail fluctuation of the compensation frame generation result of the packet loss video stream instance is determined by the frame quality fluctuation detection unit. The training error value is determined based on the deviation between the compensation frame generation result of the packet loss video stream instance and the reference video stream instance and the frame detail fluctuation. The packet loss compensation model is trained based on the training error value.

5. The method according to claim 4, characterized in that, The step of determining a training error value based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation, and training the packet loss compensation model based on the training error value, includes: The training error value of the first training stage is determined based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instances, and the frame detail fluctuation. The training error value of the first training stage is used to regulate the stability of the frame detail fluctuation. The training error value of the second training stage is determined based on the compensation frame generation result of the packet loss video stream instance, the deviation between the reference video stream instance, and the frame detail fluctuation. The training error value of the second training stage is used to set the corresponding regional attention for the regional temporal component based on the regional credibility fluctuation of the regional temporal component when determining the compensation frame generation result of the packet loss video stream instance and the deviation between the reference video stream instance. The packet loss compensation model is trained sequentially based on the training error values ​​of the first training stage and the second training stage.

6. The method according to claim 5, characterized in that, The frame detail volatility includes spatial domain frame detail volatility, temporal domain frame detail volatility, and spectral envelope feature volatility. Determining the training error value of the first training step based on the compensation frame generation result of the packet-loss video stream instance, the deviation between the reference video stream instance, and the frame detail volatility includes: Determine the compensation frame generation result of the packet loss video stream instance and the spatial domain detail deviation, temporal domain detail deviation, and spectral envelope feature deviation between the reference video stream instances; The spatial domain detail deviation is weighted according to a Gaussian distribution function with the spatial domain frame detail fluctuation as the normalization coefficient, and the temporal domain detail deviation is weighted according to a Gaussian distribution function with the temporal domain frame detail fluctuation as the normalization coefficient. The first error is determined based on the sum of the spatial domain frame detail fluctuation, the temporal domain frame detail fluctuation, the weighted spatial domain detail deviation, and the weighted temporal domain detail deviation. The spectral envelope feature deviation is weighted according to a Gaussian distribution function with the spectral envelope feature fluctuation as the normalization coefficient, and the second error is determined according to the sum of the spectral envelope feature fluctuation and the weighted spectral envelope feature deviation. The visual distortion error is determined based on the compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and the state transition frequency error is determined based on the deviation between the state transition frequency of the attention gating and the preset frequency threshold. The first error and the second error are weighted and summed to obtain the first region credibility fluctuation error. The first region credibility fluctuation error, the visual distortion error, and the state transition frequency error are weighted and summed to obtain the training error value of the first training step.

7. The method according to claim 6, characterized in that, The frame detail volatility includes spatial domain frame detail volatility, temporal domain frame detail volatility, and spectral envelope feature volatility. The step of determining the training error value for the second training stage based on the compensation frame generation result of the packet-loss video stream instance, the deviation between the reference video stream instance, and the frame detail volatility includes: The spatial domain detail deviation is weighted according to the spatial domain frame detail fluctuation, the temporal domain detail deviation is weighted according to the temporal domain frame detail fluctuation, and the third error is determined according to the sum of the weighted spatial domain detail deviation and the weighted temporal domain detail deviation. The spectral envelope feature deviation is weighted according to the spectral envelope feature fluctuation, and the weighted spectral envelope feature deviation is determined as the fourth error; The visual distortion error is determined based on the compensation frame generation result of the packet loss video stream instance and the reference video stream instance, and the state transition frequency error is determined based on the deviation between the state transition frequency of the attention gating and the preset frequency threshold. The third error and the fourth error are weighted and summed to obtain the second region credibility fluctuation error. The second region credibility fluctuation error, the visual distortion error, and the state transition frequency error are weighted and summed to obtain the training error value of the second training stage.

8. The method according to claim 1, characterized in that, The step of performing region enhancement mapping on the first temporal feature to obtain the second temporal feature includes: Based on the temporal dimension, the first temporal feature is spatially filtered in the spatial dimension to obtain the second feature map; The second feature map is standardized, and the standardized second feature map is divided based on the spatial dimension to obtain the first local relation feature and the second local relation feature. The first local relation features are standardized, and the standardized first local relation features are spatially filtered in the local region dimension based on the time sequence dimension to obtain the first feature map. The first multiplication result of the first feature map and the second local relation feature is determined. Based on the temporal dimension, the sum of the first multiplication result and the second local relation feature is spatially filtered in the spatial dimension to obtain the second temporal feature.

9. The method according to claim 1, characterized in that, The step of fusing regional features of the second temporal features to obtain the output of the temporal prediction unit located first includes: The second temporal feature is standardized, and the standardized second temporal feature is then subjected to parallel spatial filtering to obtain the third and fourth feature maps after regional feature fusion. The fourth feature map is sigmoid scaled to obtain feature selection weights; The third feature map is weighted based on the feature selection weights to obtain the output of the time-series prediction unit that is ranked first.

10. A server system, characterized in that, Includes a server, the server being used to perform the method according to any one of claims 1-9.

Images